REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  07040188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and 
reviewing  the  collection  of  information.  Send  cormients  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for 
Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  222024302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 

1.  AGENCY  USE  ONLY  Heave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

15  Dec  97 

4.  TITLE  AND  SUBTITLE 

SEE  SYMPHONY  AUTOMATED  IDENTIFICATION  AND  VISUALIZATION  OF 
EMOTIONS  IN  MUSIC 

5.  FUNDING  NUMBERS 

6.  AUTHOR(S) 

John  Robert  Billups  IV 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  AOORESS(ES) 

University  of  Colorado  at  Colorado  Springs 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

97-147 

9.  SPONSORINGfMONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

THE  DEPARTMENT  OF  THE  AIR  FORCE 

AFIT/CI,  BLDG  125 

2950  P  STREET 

WPAFB  OH  45433 

10.  SPONSORING/MONITORING 

AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION  AVAILABILITY  STATEMENT 

Unlimited  Distribution 

12b.  DISTRIBUTION  CODE 

In  Accordance  With  AFT  35-205/AFTT  Sup  1 

w  mm, 

'gL&BPOvwd  fat  gutoite 
jUfSle.  Otex&bfasoB  dais 

Kara  i  | 

release  f 

13.  ABSTRACT  (Maximum 200  words) 


19971230  158  ** 


14.  SUBJECT  TERMS 

15.  NUMBER  OF  PAGES 

101 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 

OF  REPORT 

18.  SECURITY  CLASSIFICATION 

OF  THIS  PAGE 

19.  SECURITY  CLASSIFICATION 

OF  ABSTRACT 

20.  LIMITATION  OF  ABSTRACT 

Standard  Form  298  (Rev.  2-89)  (EG) 

Prescribed  by  ANSI  Std.  239.18 

Designed  using  Perform  Pro,  WHS/DIOR,  Oct  94 


SEE  SYMPHONY 


AUTOMATED  IDENTIFICATION  AND  VISUALIZATION 
OF  EMOTIONS  IN  MUSIC 

by 

John  Robert  Billups  IV 
B.A.,  Hawaii  Pacific  University,  1990 


A  Thesis  submitted  to  the  Graduate  Faculty  of  the 
University  of  Colorado  at  Colorado  Springs 
in  partial  fulfillment  of  the  requirements  for  the  degree  of 
Master  of  Science 
Department  of  Computer  Science 


1997 


©  Copyright  By  John  Robert  Billups  IY  1997 
All  Rights  Reserved 


This  thesis  for  the  Master  of  Science  Degree  by 
John  Robert  Billups  IV 
has  been  approved  for  the 
Department  of  Computer  Science 


by 


[  M(a  (a 


Julia  Hoemer 


II  X 


Date 


IV 


To  my  son,  Noel. 

My  inspiration  for  so  many  things. 


V 


CONTENTS 

CHAPTER 

1.  INTRODUCTION . 1 

Hypothesis . 2 

Arrangement  of  the  Thesis . 2 

2.  LITERATURE  SURVEY . 4 

Properties  of  Music . 4 

Pitch . 5 

Timbre . 5 

Intensity . 6 

Duration . 6 

Music  and  Emotion . 7 

Pitch . 8 

Intensity . 9 

Duration . 9 

Relationship  of  Pitch,  Intensity,  and  Duration . 10 

Instruments  and  Pitch . 11 

Psychology  of  Color . 15 


Deafness 


17 


vi 

Eye  Music . 18 

Skin  Music . 20 

Previous  Related  Research . 21 

Color  and  Music  Association . 21 

Color  Response  to  Music . 21 

Environmaent  or  Innate . 22 

Manually  Generated  Visual  Music . 25 

Predetermined  Animations . 25 

Fantasia . 26 

Automatically  Generated  Visual  Music . 28 

Automatically  Generated  Animations . 28 

Screen  Dance . 30 

3.  METHOD . 33 

Instrument . 33 

Music . 34 

Music  to  Emotion  to  Animation . 35 

Identifying  Emotion  in  Music . 35 

Portraying  Emotion  with  Colors . 39 

Object  Shape  and  Movement . 40 

Putting  it  Together . 42 

Software  -  See  Symphony . 43 

Design . 43 


vii 

Challenges . 46 

Customizing . 47 

Running  See  Symphony . 49 

Testing . 50 

Phase  1 . 50 

Phase  n . 52 

4.  RESULTS . 54 

Questionnaire  Results . 54 

Hearing  Impaired  Results . 54 

Hearing  Results . 56 

Discussion . 58 

Color-Emotion  Association . 59 

Responses  to  Animation . 60 

General  Observations . 61 

Group  Reactions . 65 

Discussion . 65 

5.  CONCLUSIONS  AND  FURTHER  RESEARCH . 68 

Conclusions . 68 

Further  Research . 69 

REFERENCES . 71 

APPENDIX 


A.  DESCRIPTION  OF  THESIS  FOR  QUESTIONNAIRE. 


73 


viii 


B.  COVER  LETTER  AND  QUESTIONNAIRE  FOR  HEARING 


IMPAIRED  SUBJECTS  IN  RESEARCH  STUDY . 75 

C.  COVER  LETTER  AND  QUESTIONNAIRE  FOR  HEARING 

SUBJECTS  IN  RESEARCH  STUDY . 80 

D.  MESSAGES  FROM  GROUP  SUBJECTS . 85 


IX 


FIGURES 

Figure 

2. 1  Instrument  Frequency  Ranges . 12 

2.2  Adjusted  Instrument  Frequency  Ranges . 14 

2.3  Spectrograph . 15 

2.4  Animation  Terminals . 30 

2 . 5  ScreenDance  Dancer . 31 

3.1  Pitch  Shapes . 41 

3.2  Disjunct  Pitch  Shapes . 41 

3.3  Volume  Size . 42 

3.4  See  Symphony  Data  Flow  -  Level  1 . 44 

3.5  Decomposition  of  "Extract  Data"  Process  -  Level  2 . 45 

3.6  Decomposition  of  "Analyze  Data"  Process  -  Level  2 . 46 

3.7  Customize  Control  Panel . 48 

3 . 8  Individual  Presentation  Layout . 52 

3.9  Group  Presentation  Layout . 53 

4. 1  False  Fast  T empo  Identification . 62 

4.2  False  Slow  Tempo  Identification . 63 

4.3  Display  Simulation . 66 


X 


TABLES 

Table 

2. 1  Musical  Characteristics  for  Producing  Various  Emotional  Expressions . 11 

2.2  Frequencies  of  Notes . 13 

2.3  Color  and  Emotion  Associations . 17 

2.4  Predominant  Color  Choice . 23 

3. 1  Emotional  Expressions  Related  to  Musical  Elements  for  Violin . 37 

3.2  Emotion  of  Music  to  Color . 40 

3.3  Default  Color  to  Emotion  Association . 48 

3.4  Customization  Parameters . 49 

4. 1  Responses  to  Questionnaire  (Hearing  Impaired) . 55 

4.2  Color  Selection  (Hearing  Impaired) . 55 

4.3  Responses  to  Music/Animation  (Hearing  Impaired) . 56 

4.4  Responses  to  Questionnaire  (Hearing) . 57 

4.5  Color  Selection  (Hearing) . 57 

4.6  Responses  to  Music/ Animation  (Hearing) . 58 


CHAPTER  1 


INTRODUCTION 

Music  can  portray  a  wide  range  of  feelings  and  emotions.  Music  can  be  happy  or 
it  can  be  melancholy.  Music  can  be  threatening,  energetic,  or  easy  going.  For  those  of  us 
that  can  hear,  music  is  a  wonderful  medium.  Unfortunately,  there  is  a  subset  of  our 
population  who  are  not  able  to  enjoy  music  as  it  was  meant  to  be  enjoyed.  The  form  in 
which  music  is  currently  presented  deprives  the  hearing  impaired  of  the  experience  that 
so  many  of  us  take  for  granted.  The  question  to  be  asked  is  whether  music  can  be 
similarly  appreciated  via  other  senses  besides  hearing.  Is  there  a  way  to  portray  an 
emotional  qualities  of  music  for  those  without  hearing? 

In  today’s  multimedia  society,  graphics,  images,  and  video  commonly  accompany 
music  as  a  way  to  enhance  the  experience.  These  images  can  emphasize  a  feeling  or 
emotion  that  is  already  present  in  the  music.  Conversely,  the  same  images  viewed  by 
themselves  would  probably  not  portray  a  similar  emotion. 

Music  can  be  broken  down  into  four  basic  properties:  pitch,  timbre,  intensity,  and 
duration  (Christ  et  al,  1980).  The  characteristics  of  these  four  properties  determine  the 
mood  or  emotion  of  the  music  (Politoske,  1979).  For  a  visual  medium  to  mirror  the  aural 
nature  of  music  the  four  properties  must  be  reflected. 
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Today,  the  computer  is  a  common  part  of  everyday  life  in  many  ways.  Computers 
can  read,  play,  and  analyze  music.  Computers  can  also  generate  impressive  graphics. 
Putting  these  two  attributes  together  will  enable  computers  to  automatically  detect  and 
translate  the  aural  properties  of  music  into  visual  properties. 

Simply  stated,  the  emotions  of  music  can  be  automatically  identified  and 
displayed  visually  in  an  animated  fashion  using  color,  shape,  and  movement.  Since 
music  is  a  complex  medium  this  work  will  concentrate  on  solo  violin  compositions. 

Music  is  deeply  embedded  into  our  culture  and  in  many  other  cultures  of  the 
world.  By  providing  an  additional  medium,  those  who  can  not  hear  music  could  see 
music  and  therefore  achieve  a  broader  experience  in  everyday  life. 

Hypothesis 

It  is  possible  to  automatically  identify  the  emotions  portrayed  by  music  and 
display  those  same  emotions  visually  through  the  use  of  computer  hardware  and 
software. 

Arrangement  of  the  Thesis 

This  thesis  deals  with  a  wide  range  of  topics.  Most  of  the  areas  are  not  computer 
science  related,  but  they  must  be  covered  as  a  foundation  for  the  work  that  is 
accomplished  in  this  thesis  and  for  the  work  that  will  follow. 

Chapter  2  is  the  literature  review  of  topics  relevant  to  this  thesis.  These  topics 
include  :  properties  of  music,  music  and  emotion,  instruments  and  pitch,  psychology  of 
color,  deafness,  and  previously  related  work  in  the  field  of  manually  and  automatically 
generated  visual  music.  Chapter  3  deals  with  the  method  used  to  show  that  the 
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hypothesis  is  possible.  Chapter  3  covers  the  following:  instrument  and  music  used, 
translation  from  music  to  emotion  to  animation,  and  software  implementation.  The 
software  section  of  Chapter  3  also  describes  how  the  software  was  tested  with  deaf  and 
hearing  individuals.  The  results  of  the  methods  are  detailed  in  Chapter  4.  Conclusions 
and  suggestions  for  further  research  are  discussed  in  Chapter  5.  Appendices  A,  B,  and  C 
contain  descriptions,  cover  letters,  and  questionnaires  used  in  the  testing  of  the  software 
with  deaf  and  hearing  subjects.  Appendix  D  contains  messages  from  six  hearing  subjects 
involved  in  the  testing  of  the  software. 


CHAPTER  2 


LITERATURE  REVIEW 

This  work  crosses  multiple  disciplines  and  areas:  properties  of  music,  music  and 
emotion,  psychology  of  color,  deafness,  and  software  engineering.  I  researched  each  area 
thoroughly  enough  to  gain  the  knowledge  necessary  to  achieve  my  goal  of  translating  the 
emotion  of  music  to  color,  shape,  and  movement. 

Properties  of  Music 

Musical  sound  can  be  broken  down  into  four  basic  properties:  pitch  (frequency), 
timbre  (tone  quality),  intensity  (loudness),  and  duration  (length  of  time  note  is  held) 
(Christ  et  al,  1980).  Pitch  ranges  from  high  to  low.  Intensity  ranges  from  loud  to  soft. 
Duration  ranges  from  long  to  short.  Timbre,  the  characteristic  of  the  sound,  depends  on 
the  instrument  being  played.  All  four  properties  can  be  measured  and  analyzed. 

Music  contains  many  more  properties  and  elements  than  those  I  have  listed,  such 
as  melody,  rhythm,  harmony  and  accent  to  name  a  few.  From  my  readings  it  appears  to 
me  that  music’s  other  characteristics  can  be  broken  down  into  the  previously  mentioned 
four  basic  properties. 
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Pitch 

The  number  of  vibrations  per  second  determines  the  pitch  of  a  given  musical 
tone  (Politoske,  1979).  The  faster  the  vibrations  the  higher  the  pitch.  The  human  ear,  on 
average,  can  hear  frequencies  between  20  and  20,000  cycles  per  second  (cps)  (Culver, 
1956). 

From  note  to  note  over  a  short  term  there  are  a  finite  number  of  ways  that  the 
pitch  can  change.  The  pitch  can  remain  steady,  go  higher,  go  lower,  or  alternate.  If  the 
pitch  rises  continually  then  it  is  ascending.  If  the  reverse  is  true  then  it  is  descending.  If 
there  is  a  large  distance  between  consecutive  pitches  then  they  are  disjunct;  if  the 
distance  is  small  they  are  conjunct  (Politoske,  1979).  When  two  or  more  frequencies  are 
invoked  simultaneously  and  produce  an  unpleasant  sound  it  is  said  to  be  dissonant;  if  the 
effect  is  pleasing  it  is  consonant  (Culver,1956).  The  direction  in  which  the  pitch  travels 
has  a  major  impact  on  which  emotions  are  being  expressed  in  the  music. 

Timbre 

The  timbre  of  the  music  is  simply  the  quality  of  the  music  (Culver,  1956).  Timbre 
is  what  lets  us  distinguish  between  instruments.  Each  instrument  has  its  own  voice;  they 
each  have  their  own  particular  sound.  For  example,  we  can  distinguish  between  a  string 
instrument  and  a  woodwind  instrument  playing  the  same  note  at  the  same  intensity  level. 

At  this  point  in  my  work,  timbre  will  not  be  much  of  a  factor  since  I  will  be 


focusing  on  one  instrument. 
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Intensity 

Intensity  is  the  loudness  of  the  music.  A  change  in  volume  can  occur  quickly  or 
slowly  depending  on  the  effect  that  is  desired.  As  with  pitch  there  are  a  finite  number  of 
things  that  can  happen  with  the  volume  in  a  musical  piece  from  note  to  note.  The 
volume  can  remain  steady,  increase,  decrease,  or  alternate. 

In  relation  to  hearing,  the  intensity  level  of  a  given  sound  must  change  by  one 
decibel  (26%)  before  the  human  ear  can  detect  a  change  in  the  strength  of  that  sound 
(Culver, 1956). 

In  musical  works  the  level  of  loudness  has  been  classified  into  following 
categories:  very  soft,  soft,  moderately  soft,  moderately  loud,  loud,  very  loud,  sudden 
stress  on  a  single  note,  and  loud  followed  suddenly  by  soft  (Politoske,  1979). 

Again,  as  with  pitch,  the  direction  and  level  of  the  volume  has  major  impact  on 
which  emotions  are  being  expressed  in  the  music. 

Duration 

Duration  is  the  length  of  time  that  a  note  is  held.  Different  durations  are  specified 
as  follows:  whole,  half,  quarter,  eighth,  sixteenth,  thirty-second,  and  sixty-fourth 
(Politoske,  1979).  The  actual  length  or  number  of  beats  that  each  duration  represents 
depends  on  how  many  beats  a  whole  note  represents.  If  a  whole  note  is  four  beats  then 
each  subsequent  duration  is  the  given  fraction  of  the  whole  note.  For  example,  in  this 
case  a  quarter  note  would  be  one  beat. 
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The  duration  of  silence,  the  length  of  time  between  notes,  is  also  an  important 
factor.  The  combination  of  duration  of  note  and  silence  (which  may  be  none)  make  up 
the  tempo  of  the  music. 

As  with  pitch  and  intensity,  the  duration  from  note  to  note  can  either  remain 
constant,  become  shorter,  become  longer,  or  alternate. 

It  should  be  obvious  here  that  the  time  that  a  note  is  held  also  has  a  significant 
impact  on  the  feeling  of  the  music. 

Music  and  Emotion 

“Those  who  appreciate  music  do  not  listen  passively  and  passionlessly.  They 
actively  listen  for  what  is  to  come  in  light  of  what  has  been  foretold,  pointed 
ahead  by  the  unfolding  progressions  (Goldman,  1995)” 

Each  of  the  three  elements,  pitch,  intensity,  duration  contribute  to  emotions  and 
feelings  portrayed  by  music.  While  each  has  its  influence,  it  is  most  often  not  a  factor  of 
just  one  of  the  properties.  It  is  more  likely  a  combination  of  two  or  more  that  brings 
music  to  life. 

It  should  be  noted  here  that  there  are  two  major  camps  regarding  music  and 
emotion:  cognitivist  and  emotivist  (Radford,  1989).  Cognitivists  believe  that  music  may 
express  various  emotions  such  as  sadness,  fear,  anger,  happiness,  etc.,  and  that  we  can 
identify  these  emotions  in  music.  They  claim  that  listening  to,  for  example,  sad  music 
does  not,  in  turn,  make  us  sad.  On  the  other  hand,  emotivists  say  that  if  the  music  is  sad 
then  it  may  make  us  feel  so.  Emotivists  argue  that  if  the  music  does  not  move  you  to  a 
certain  emotion  than  the  music  should  not  be  labeled  as  such.  I  will  not  get  involved  in 
that  argument.  I’m  making  the  assumption  that  music  does  portray  and  invoke  emotions. 
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What  Pm  most  interested  in  is  the  portrayal  of  emotions  in  music.  There  is  greater 
agreement  on  what  emotions  music  portrays  than  on  what  emotions  music  evokes  (Storr, 
1992).  The  mood  of  the  individual  or  the  occasion  are  just  two  factors  which  may  affect 
a  person’s  reaction  to  the  same  piece  of  music  (Storr,  1992). 

Another  point  of  interest  is  that  the  degree  to  which  music  affects  us  depends  a 
great  deal  on  how  we  perceive  it.  The  difference  comes  from  whether  we  hear  music  or 
listen  to  music  (Cook,  1990).  The  difference  is  how  much  attention  is  paid  to  the  music. 

I  think  it  is  simple  to  agree  that  a  piece  of  music  played  in  a  concert  hall  or  in  one’s 
living  room  while  relaxing  (listening)  has  a  much  more  profound  effect  on  ones  emotions 
than  does  the  same  music  being  played  as  background  music  in  a  shopping  mall 
(hearing).  When  I  discuss  the  emotions  in  music  I  am  referring  to  listening  as  opposed 
to  just  hearing  the  music. 

The  emotional  impact  of  each  element  will  be  described  below,  followed  by  a 
discussion  of  the  effect  of  the  elements  combined. 

Pitch 

Pitch,  by  itself,  is  strongly  linked  to  perceived  emotions  (Bruner,  1990).  The 
higher  the  pitch  the  more  exciting  or  happy  the  music  seems  to  become.  On  the  other 
hand,  a  low  pitch  seems  to  represent  sadness. 

Pitch  can  also  be  thought  of  movement  in  spatial  terms.  Ascending  pitches 
produce  a  feeling  of  expansion,  excitement,  or  upward  movement;  descending  pitches 
give  the  feeling  of  falling  or  a  downward  pull  that  may  be  associated  with  rest,  or  finality 
(Politoske,  1979).  An  alternating  trend  in  pitch  would  give  the  feeling  of  moving  up  and 
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down.  The  height  and  depth  of  the  movements  would  greatly  depend  on  whether  the 
pitches  are  conjunct  or  disjunct  and  the  length  of  ascent  and  descent. 

Disjunct  tunes  are  more  likely  to  be  dramatic,  energetic,  and  agitated,  while 
conjunct  tunes  are  sweeter,  playful,  happy  and  more  lyrical  (Bruner,  1990;  Politoske, 

1 979).  The  same  can  be  said  for  dissonant  and  consonant  sounds,  respectively. 

Intensity 

Intensity  has  a  definite  link  to  the  mood  of  a  musical  piece.  Loud  music  is 
animated,  triumphant,  or  exciting,  while  softer  music  is  characterized  as  delicate, 
tranquil,  peaceful,  or  serious  (Bruner,  1990).  It  is  clear  that  the  amount  of  volume  can  be 
related  to  the  amount  of  energy  that  the  musical  piece  is  portraying.  The  louder  a  piece 
the  more  energy  the  piece  portrays  to  the  listener. 

Duration 

I  did  not  find  much  written  about  duration  and  its  direct  relation  to  emotions. 
However,  there  is  information  on  tempo  of  music.  As  discussed  previously,  tempo  is 
simply  a  combination  of  durations  of  notes  and  silence.  For  example,  a  musical  piece 
with  short  duration  on  notes  and  silence  would  represent  a  quick  tempo.  If  the  duration 
of  either  were  increased  the  tempo  would  decrease. 

Looking  at  how  tempo  portrays  emotions  it  seems  almost  natural  to  imply  that  a 
quick  tempo  represents  quickening,  excitement,  or  activity  (Politoske,  1979).  It  is  equally 
natural  to  suggest  that  a  slow  tempo  can  be  sad,  serious,  or  serene  (Bruner,  1990). 
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Relationship  of  Pitch,  Intensity,  and  Duration 

Each  of  the  three  properties  (pitch,  intensity,  and  emotion)  have  emotional 
implications  of  their  own,  but  individually  they  do  not  make  up  a  whole  emotion.  The 
true  effect  comes  when  they  are  combined. 

An  interesting  observation  discussed  by  Goldman  (1995)  is  that  the  main  basis  for 
our  recognition  of  pitch,  intensity,  and  duration  in  music  is  due  to  their  resemblance  of 
natural  human  expressions  of  emotion  by  voice,  demeanor,  and  behavior.  He  goes  on  to 
say,  if  we  accept  a  spatial  description  of  notes  then  we  can  also  describe  it’s  movement 
which,  can  then  be  compared  to  human  movements  and  natural  expressions  of  emotion. 

Let’s  apply  this  thinking  to  some  common  emotional  states.  When  one  is  sad, 
movements  tend  to  be  slower;  the  voice  is  lower,  softer,  and  drawn  out.  Even  cries  of 
sadness  (in  Western  culture)  are  typically  low  in  volume  and  lacking  in  energy.  This  fits 
what  was  previously  described  as  sad  with  the  three  properties,  the  pitch  is  low,  intensity 
is  low,  and  duration  is  long. 

When  one  is  happy,  movements  are  quicker,  the  voice  is  higher,  faster,  louder, 
and  full  of  energy.  Laughter  is  a  good  example  of  these  attributes.  Happy  music  has  the 
same  general  properties:  high  pitch,  fast  tempo,  medium  intensity. 

Table  2. 1  list  a  compilation  of  musical  elements  and  their  relation  in  portraying 


different  emotions. 
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Table  2. 1  Musical  Characteristics  for  Producing  Various  Emotional  Expressions 


Musical 

Element 

Serious 

Sad 

Emotional 

Serene 

Expression 

Happy 

Exciting 

Frightening 

Mode 

Major 

Minor 

Major 

Major 

Major 

Minor 

Tempo 

Slow 

Slow 

Slow 

Fast 

Fast 

Slow 

Pitch 

Low 

Low 

Medium 

High 

Medium 

Low 

Rhythm 

Firm 

Firm 

Flowing 

Flowing 

Uneven 

Uneven 

Harmony 

Consonant 

Dissonant 

Consonant 

Consonant 

Dissonant 

Dissonant 

Volume 

Medium 

Soft 

Soft 

Medium 

Loud 

Varied 

Source:  Data  from  Gordon  Bruner,  “Music,  Mood,  and  Marketing.”  Journal  of  Marketing  54,  no.  4 
(1990):  100. 


As  can  be  seen,  to  accurately  determine  the  feelings  that  a  piece  of  music  is 
portraying  its  imperative  to  examine  multiple  properties  simultaneously.  While 
individual  elements  of  music  do  have  emotional  connotations,  they  do  not  by  themselves 
tell  the  whole  story. 

Instruments  and  Pitch 

A  healthy  human  ear  can  hear  sounds  that  are  in  the  range  of  20  to  20,000cps. 

The  frequency  range  produced  by  musical  instruments  is  approximately  27  to  4200cps 
(Backus,  1977).  Figure  2.1  displays  the  frequency  ranges  of  some  popular  musical 
instruments. 

The  piano  is  the  only  instrument  listed  that  can  obtain  the  full  spectrum  of 
frequencies  that  can  be  produced  by  musical  instruments.  It  is  clear  that  the  piano  can 
reach  all  notes  from  high  to  low.  The  other  instruments  listed  can  only  obtain  a  subset  of 
the  frequency  range. 
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Figure  2. 1  Instrument  Frequency  Ranges 

piccalo  I  1 

flute  I  1 

°b°e  |  I 

soprano  clarinet  1  I 

basson  1  I 

trumpet  I  I 
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Sources:  Data  from  Backus,  John.  The  Acoustical  Foundations  of  Music.  New  York:  W.  W.  Norton  & 
Company,  1977;  White,  Harvey  and  Donald  White.  Physics  and  Music  -  The  Science  of  Musical  Sound. 
Philadelphia:  Saunders  College,  1980;  and  Culver,  Charles.  Musical  Acoustics,  4th  ed.  New  York: 
McGraw  Hill,  1956. 


What  is  the  definition  of  a  high,  medium,  or  low  pitch?  The  International 
Standards  Organization  recommended,  in  1953,  that  440cps  be  used  as  the  standard 
frequency  for  music  worldwide  (White  and  White  1980).  Today,  this  is  what  most 
musicians  in  symphony  orchestras  use  to  tune  their  instruments. 

Examining  Figure  2.1,  most  of  the  instruments  have  440cps  within  their 
frequency  ranges.  More  specifically,  the  violin’s  frequency  range  is  approximately  170 
to  3000cps.  Since  440cps  is  the  standard,  can  we  consider  it  to  be  the  medium  pitch  for 


orchestral  music? 
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Table  2.2  Frequencies  of  Notes 


c, 

32 

C2 

65 

c3 

132 

c4 

264 

C5 

528 

C6 

1056 

C7 

2112 

D, 

36 

D2 

73 

D3 

148 

d4 

297 

Ds 

594 

D6 

1188 

d7 

2376 

E, 

41 

e2 

82 

e3 

165 

e4 

330 

e5 

660 

e6 

1320 

e7 

2640 

Fi 

43 

f2 

87 

f3 

176 

f4 

352 

f5 

704 

f6 

1408 

f7 

2816 

G, 

48 

g2 

97 

g3 

198 

g4 

396 

g5 

792 

g6 

1584 

g7 

3168 

Ai 

55 

A2 

110 

a3 

220 

A4 

440 

a5 

880 

K 

1760 

a7 

3520 

Bi 

61 

b2 

123 

b3 

247 

b4 

495 

b5, 

990 

b6 

1980 

b7 

3960 

Sources:  Data  from  Backus,  John.  The  Acoustical  Foundations  of  Music.  New  York:  W.  W.  Norton 
&  Company,  1977  and  White,  Harvey  and  Donald  White.  Physics  and  Music  -  The  Science  of 
Musical  Sound.  Philadelphia:  Saunders  College,  1980. 

Notes:  The  decimal  points  have  been  dropped  for  readability.  The  outermost  ranges  of  the  scale  (Co- 
Bo  and  Cg-Bg)  are  not  shown  because  they  contain  values  that  are  not  obtainable  by  popular 
instruments.  The  letters  are  for  the  notes  of  the  scale.  The  subscript  number  next  to  the  notes 
designate  the  octave  of  the  note. 


Looking  at  how  440cps  (A4)  places  in  the  scale  of  notes  (Table  2.2)  we  see  that  it 
represents  the  fourth  octave  and  it  falls  right  in  the  middle.  From  this  example,  I  will 
assume  that  notes  below  the  scale  of  A4  are  low  notes  and  the  notes  above  the  scale  of  A4 
are  high  notes.  The  degree  to  which  these  notes  are  high  or  low  depends  on  their 
distance  from  A4. 

Further  examination  of  Table  2.2  also  reveals  that  the  frequencies  of  musical 
notes  are  not  linear.  That  is,  the  spacing  between  the  notes  of  each  octave  doubles  as  you 
travel  up  the  scale.  For  example,  the  difference  between  A3  and  A4  is  220cps  while  the 
distance  between  A4  and  A5  is  440cps.  Figure  2.2  gives  a  more  accurate  depiction  of  the 
frequency  ranges  of  the  selected  instruments  and  how  they  fall  in  the  musical  scale.  This 
view  also  makes  it  easier  to  see  how  A4  relates  to  instruments  ranges. 
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Figure  2.2  Adjusted  Instrument  Frequency  Ranges 
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Sources:  Data  from  Backus,  John.  The  Acoustical  Foundations  of  Music.  New  York:  W.  W.  Norton  & 
Company,  1977;  White,  Harvey  and  Donald  White.  Physics  and  Music  -  The  Science  of  Musical  Sound. 
Philadelphia:  Saunders  College,  1980;  and  Culver,  Charles.  Musical  Acoustics f  4th  ed.  New  York: 
McGrawHill,  1956. 


Viewing  a  spectrograph  (stereo  equalizer  display)  while  listening  to  orchestral 
music  strengthens  my  assumption.  I  played  several  pieces  of  music  (primarily  violin 
concertos)  on  my  CD  player  and  kept  close  tabs  on  what  frequencies  where  peaking 
(Figure  2.3  is  a  close  approximation  of  what  the  display  looks  like,  the  numbers  along  the 
bottom  represent  frequencies).  When  there  were  notes  that  I  perceived  as  being 
predominately  low,  the  bars  to  the  left  of  500  were  the  most  active.  Likewise,  when  I 
perceived  notes  to  be  high,  the  bars  right  of  500  were  most  active.  Overall,  the  500  bar 


was  the  most  active. 
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Figure  2.3  Spectrograph 


30  60  125  250  500  IK  2K  4K  8K  16K 


Closer  examination  of  Figure  2.2  reveals  that  the  frequency  ranges  of  some 
instruments  (piccolo,  bass  tuba,  and  bass  violin)  do  not  include  A4.  Does  this  mean  that 
those  instruments  can  only  be  classified  as  high  or  low,  depending  on  which  side  of  A4 
they  fall?  When  these  instruments  are  heard  from  within  an  orchestra  they  do  sound 
relatively  high  or  low,  but  what  about  when  the  instrument  is  heard  solo?  A  piccolo  that 
played  a  scale  from  B5  (990cps)  to  E7  (2640cps)  will  surely  give  the  impression  of  going 
from  a  low  to  a  high  note,  but  that  entire  range  is  high.  Should  the  determination  of  high 
and  low  frequencies  be  made  on  the  frequency  range  of  music  or  the  frequency  range  of 
the  specific  instrument?  Unfortunately  this  topic  is  beyond  the  scope  of  this  study  and  is 
reserved  for  further  research. 

Psychology  of  Color 

My  main  interest  in  color  lies  in  what  emotions  different  colors  portray.  I  hope  to 
use  color  to  play  a  large  part  in  representing  the  emotive  qualities  of  music.  As  with 
music,  I  am  aware  that  color  alone  can  not  fully  characterize  emotion,  but  I  believe  it  is  a 
major  piece  of  the  puzzle. 
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As  with  music,  color  can  also  be  broken  down  into  basic  properties.  The  three 
elements  of  color  are  as  follows:  hue  (perceived  color),  brightness  (dull  to  sharp  and 
clear),  and  saturation  (shallow  to  deep  and  strong)  (Sharpe,  1974). 

Colors  have  different  meaning  to  many  cultures.  When  I  discuss  a  color  having  a 
certain  emotion  or  feeling  associated  with  it  I  am  referring  to  American  preferences.  For 
a  quick  example,  black  is  a  color  of  mourning  in  our  culture,  but  the  opposite  is  true  in 
some  eastern  cultures,  where  white  is  a  color  of  mourning. 

There  have  been  a  number  of  studies  that  have  set  out  to  investigate  how  people 
correlate  emotions  and  colors.  After  examining  the  results  of  such  studies  and/or 
conducting  their  own  research  Birren  (1961)  and  Sharpe  (1974)  came  to  the  following 
general  conclusions  as  noted  in  Table  2.3. 

The  results  overlap  in  places  and  are  not  conclusive,  but  a  general  remark  can  be 
made  of  the  data  in  whole.  The  warmer  colors  (red  -  yellow)  are  more  energetic  and 
happy,  while  the  cooler  colors  (green  purple)  are  more  peaceful  and  calm. 
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Table  2.3  Color  and  Emotion  Associations 


Color 

Birren 

Sharpe 

Red 

Passionate,  exciting. 

Excitement, 

active 

stimulation, 

aggression 

Orange 

Jovial,  lively, 

Excitement, 

energetic,  forceful 

stimulation, 

aggression 

Yellow 

Cheerful,  inspiring, 
vital,  celestial 

Cheer,  gaiety,  fun. 

Green 

Quieting,  refreshing. 

Calm,  security. 

peaceful 

peace 

Blue 

Subduing, 

Calm,  security. 

melancholy,  sober 

peace 

Purple 

Dignified,  pompous. 

Dignity,  royalty. 

mournful,  mystic 

sadness 

White 

Pure,  clean,  frank, 

Purity,  innocence. 

youthful 

holiness 

Black 

Ominous,  deadly. 

melancholy. 

depressing 

sadness,  depression 

Sources:  Data  from  Faber  Birren.  Color  Psychology  and  Color  Therapy  -  A  Factual  Study  of 
Color  on  Human  Life .  New  York:  University  Books,  Inc.,  1961.;  Deborah  Sharpe.  The 
Psychology  of  Color  and  Desigti.  Chicago:  Nelson-Hall  Company,  1974. 


Deafness 

The  essence  of  this  work  is  to  provide  an  additional  medi  um  to  which  hearing 
impaired  persons  may  achieve  a  fuller  enjoyment  from  music.  I  do  not  know  what  it  is 
like  to  be  deaf  or  hearing  impaired.  The  goal  of  my  research  in  this  area  is  gain  some 
understanding  of  deaf  perceptions. 

Among  the  hearing  impaired,  there  are  relatively  few  with  absolutely  no  sense  of 
hearing;  patterns  of  hearing  loss  can  be  very  complex  (Higgins,  1980).  There  is  a  vast 
range  of  hearing  impairment.  The  difference  lies  in  what  frequency  range  they  can 
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perceive.  For  example,  the  ear  can  perceive  sounds  between  20  and  20,000  cps,  human 
speech  is  in  the  range  of  250  to  4000  cps.  Losing  a  relatively  small  portion  of  that  range 
could  render  an  individual  incapable  of  hearing  spoken  words.  My  concern  here  is  the 
perception  of  music.  Obviously,  musical  instruments  can  traverse  a  larger  frequency 
range  than  the  human  voice.  Since  there  is  no  universal  pattern  of  hearing  loss  I  will  not 
focus  on  any  particular  frequency  range.  Also,  when  I  make  or  refer  to  statements  about 
the  capabilities  of  deaf  people  I  am  aware  that  these  statements  do  not  apply  to  all  deaf 
people. 

Because  deaf  people  cannot  hear  music  it  does  not  mean  that  they  cannot 
appreciate  such  musical  concepts  as  harmony,  dissonance,  and  variation  (Padden  and 
Humphries,  1988).  There  are  three  different  types  of  music:  ear  music,  eye  music,  and 
skin  music  (Zak,  1997).  Through  the  eyes  and  skin,  many  of  the  properties  of  music  can 
be  learned  and  appreciated. 

Eye  Music 

Eye  music  is  observance  of  motion  that  mirrors  properties  of  music.  There  is 
movement  in  everyday  life  that  can  be  thought  of  as  rhythmic,  but  a  better  example  of 
eye  music  is  poems  expressed  in  sign  language.  The  movement  of  the  signs  and  the  body 
language  can  communicate  various  musical  properties.  The  following  is  a  description  by 
Padden  and  Humphries  (1988)  of  a  sign  reading  of  a  poem,  “Windy,  Bright  Morning” 
written  and  performed  by  Clayton  Valli  (1985): 

The  poem  describes  the  edge  of  wakefulness  and  sleep  and  begins  with  a 
window  and  an  external  force: 
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Through  the  open  window 

with  its  shade  swinging,  sunshine,  playful, 

taps  my  sleepy  eyes. 

The  hand,  used  to  represent  the  shade,  moves  in  a  slightly  irregular  but  not 
unpleasant  rhythm. 

Breezes  dance  in  my  room, 
around  my,  not  shy,  but  gentle, 
letting  me  know  that  it’s  time 
to  get  up!  Slowly  I  wake, 
my  eyes  stung  by  the  sunlight 
flashing  past  the  swinging  shade 
that  seems  to  know  I’m  deaf. 

The  presence  of  the  light  is  unmistakable;  the  movement  revolves  around  the 
center  of  the  light 

I  stand  up,  tired,  ignoring  the  light 
chilled  in  the  dancing  air 
that  meets  me  by  the  window 
I  closely  shut  it.  And  with  the  shade  still, 
my  room  darkens. 

The  irregular  movement  abruptly  ceases,  and  the  room  becomes  silent.  As 
Valli  moves  back  to  the  familiar  bed,  movement  is  slow  and  comforting. 

Happy 

back  under  the  covers, 

I’m  drowsy,  purring,  warm  .  .  . 

The  audience,  lulled  by  Valli’s  slow  delivery,  is  unprepared  for  the  next  verse: 

But,  suddenly,  how  strange! 

The  shade  flaps  widely, 
bright,  dark,  bright,  dark,  bright 
Fierce  wind  flung  open  the  window  . . . 
so  bitter  cold,  so  cold,  the  wind,  the  shade, 
the  storm! 

The  movement  is  wild  and  unpredictable.  Valli  as  experiencer  widens  his  eyes 
and  moves  his  body  with  a  sense  of  urgency. 

Slowly  I  rise,  and  try  to  make  them  calm  down. 

As  he  moves  toward  the  window,  the  window,  fonnerly  dissonant,  changes 
again,  beginning  to  come  together  in  one  organized  and  focused  form: 
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The  wind,  the  shade,  dancing  gracefully,  happy. 

One  bright  ray  gently  pulls  me 
to  raise  up  the  shade 
like  unwrapping  a  gift. 

Warm  sunlight  tickles  me, 
morning  breeze  laughs  with  me . . . 

Joyful,  I  welcome  the  day. 

(Valli,  1985) 

The  feeling  of  dissonance,  consonance,  and  tempo  can  easily  be  identified  in  this 
poem.  The  changes  in  energy  are  also  representative  of  changes  in  moods.  This  is  yet 
another  example  of  how  body  movement  can  be  described  with  musical  terms  which  can 
lead  to  a  musical  description  of  emotions.  This  also  supports  the  thought  that  such 
musical  properties  can  be  appreciated  visually. 

Skin  Music 

Skin  music  is  feeling  music  via  the  vibrations  through  the  air.  Even  people  who 
are  extremely  deaf  can  still  feel  the  vibrations  of  music  through  speakers  (air)  or  the  floor 
(Zak,  1997).  This  makes  perfect  sense  because  sound  is  nothing  more  than  vibrations  in 
the  air.  Depending  on  the  pitch  and  volume  those  vibrations  can  be  felt  upon  the  skin,  in 
other  words,  if  it’s  loud  enough  and  has  enough  bass. 

Professor  Daniel  M.  Berry  of  Technion  University  in  Israel  (Berry,  1997)  told  me 
that  although  he  is  nearly  deaf,  he  hears  rhythms  very  well,  so  well  in  fact  that  he  is  able 
to  dance.  He  also  enjoys  rap  music  because  of  it’s  strong  beat. 

This  is  evidence  that  there  is  more  than  one  way  to  appreciate  the  properties  of 
music.  The  absence  of  hearing  does  not  totally  deprive  the  Deaf  of  an  aural  experience. 
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Previous  Related  Research 

Color  and  Music  Association 

Synesthesia  is  a  psychological  phenomenon  in  which  people  have  the 
simultaneous  response  in  more  than  one  sensory  mode  (Radocy  and  Boyle,  1979). 
Chromesthesia,  a  specific  form  of  synesthesia,  is  a  condition  where  audible  sounds 
create,  simultaneously  ,  visual  color  as  well  and  aural  sensation  (Cuteitta  and  Haggerty, 
1987).  Being  able  to  associate  color  with  music  or  any  other  audible  sensations  does  not 
constitute  chromesthesia  since  the  connection  to  color  comes  after  the  sound. 

Color  Response  to  Music 

In  a  study  conducted  by  Karwoski  and  Odbert  (1938),  involving  148  college 
students,  approximately  60  percent  (87  students)  had  a  color  response  to  music.  Of  the 
87  students,  65  percent  experienced  chromesthesia,  88  percent  associated  a  color,  and  5 1 
percent  felt  a  color  response.  They  came  to  the  conclusion  that  a  majority  of  the 
population  has  some  kind  of  association  between  color  and  music. 

Karwoski  and  Odbert  (1938)  also  noted  the  following  findings:  blue  is  associated 
with  slow  music,  red  is  associated  with  fast  music,  light  colors  with  high  notes,  dark 
colors  with  deep  notes.  Aside  from  red  and  blue  there  was  no  mention  of  the  impact  of 
other  colors  or  if  the  test  subjects  came  up  with  the  same  colors.  Another  interesting  part 
of  the  study  was  that  they  found  that  the  vertical  dimension  is  related  to  pitch  and  that 
intensity  may  be  denoted  by  depth. 
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Environment  or  Innate 

The  question  to  ask  is  whether  color  associations  are  innate  or  are  acquired  and 
learned  from  our  individual  environments.  A  study  conducted  by  Cutietta  and  Haggerty 
(1987)  sought  to  answer  that  question.  They  believed  that  there  would  be  two  possible 
conclusions.  One,  if  music/color  association  is  a  product  of  environment  then  you  would 
expect  a  strengthening  of  this  association  throughout  the  school  years.  Two,  if  music 
/color  associations  are  due  to  early  synesthesia  then  you  would  expect  a  constant 
response  throughout  the  school  years,  perhaps  getting  weaker  as  one  matures. 

The  study  was  conducted  over  three  phases.  The  first  phase  involved  subjects 
from  elementary  school,  secondary  school  and  college.  Phase  two  involved  subjects  ages 
3  to  1 1 .  Phase  three  involved  subjects  from  college  to  senior  citizens.  All  subjects 
listened  to  three  different  30-second  examples  of  taped  music.  The  example  were:  (1) 
Gustav  Holst’s  Suite  No.  1  in  E\  third  movement,  “March,”  measures  1-36;  (2)  Modest 
Moussorgsky’s  Pictures  at  an  Exhibition,  fourth  movement,  “Bydlo,”  measures  1-20;  and 
(3)  George  Frideric  Handel’s  Music  for  the  Royal  Fireworks,  “Bourree,”  measures  1 1-26. 
Each  musical  piece  was  chosen  because  of  its  distinct  and  separate  qualities.  The 
“March”  is  majestic  and  vigorous,  the  “Bydlo”  is  plodding  and  laboring,  and  the 
“Bourree”  is  lively  (Cuteitta  and  Haggerty,  1987).  From  here  forward  any  reference  to 
the  music  examples  will  be  the  names  in  quotations. 

Each  subject  listened  to  a  music  example  and  indicated  which  color  they 
associated  with  the  music.  The  subject  were  given  a  standard  color  wheel  with  the 
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following  colors:  orange,  yellow,  green,  blue,  purple,  and  red.  The  predominant  color 
choices  for  each  phase  are  listed  in  Table  2.4. 


Table  2.4  Predominant  Color  Choice 


Phase  1  (Grades  1 -college) 

Phase  2  (Ages  3-11) 

Phase  3  (college  -  70s) 

“March” 

Red,  all 

Blue  3,  4,  9,  10 

Purple  7 

Red  5,6,8,  11 

Orange  6 

Yellow  8 

Green  4 

Red  College  -40s,  60s,  70s 
Yellow  50s 

“Bydlo” 

Blue,  1-6, 8-college 

Purple,  7 

Blue  9,  10,  11 

Purple  3,  7 

Red  5,  7,  8 

Orange  6 

Yellow  4,  7 

Green  (none) 

Blue  College,  20s,  50s,  70s 
Purple  30s,  40s,  60s 

“Bourree” 

Yellow  1 -2,4,6-college 
Orange,  3  and  5 

Blue  5,  8 

Purple  4 

Red  3,  9 

Orange  7 

Yellow  6,  10,  11 

Green  11 

Yellow  College  -  40s,  60s, 
70s 

Red  50s 

Source:  Data  from  Robert  Cutietta  and  Kelly  Haggerty.  “A  Comparative  Stud 

ty  of  Color  Association  with 

Music  at  Various  Age  Levels.”  Journal  of  Research  in  Music  Education ,  35,  no.  2  (1987):  84-88. 
Note:  If  an  age  indicator  is  listed  twice  that  means  there  was  a  tie  for  that  age  group 


There  is  consistency  across  age  groups  in  phases  one  and  three.  The  results  for 
phase  two  appears  to  be  random,  but  the  colors  selected  in  the  other  phases  still  have  the 
highest  concentrations  for  each  musical  example.  No  reasons  were  given  for  the 
scattered  results  in  phase  two. 

Cuteitta  and  Haggerty  (1987)  discuss  that  the  theory  of  chromesthesia  can  not  be 
supported  by  phase  one  due  to  the  inconsistencies  with  “Bydlo”  and  “Bourree.” 
Examining  the  differences  in  color  selection  of  the  two  musical  pieces  one  could  argue 
that  there  are  no  inconsistencies  worth  mentioning.  Recalling  Table  2.3,  one  will  notice 
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that  the  emotional  responses  to  blue  and  purple  are  very  similar,  as  well  are  the  responses 
to  orange  and  yellow. 

The  discussion  that  follows  phase  one  mentions  that  music  similar  to  the  styles  of 
“March”  and  “Bydlo”  are  commonly  used  in  media  that  is  targeted  towards  children, 
while  Baroque  style  music  like  “Bourree”  is  not.  They  say  that  this  explains  the 
inconsistencies  in  phase  one.  Rather,  my  speculation  is  that  it  explains  the 
inconsistencies  in  phase  two.  Phase  two  looks  like  confusion.  Children  are  bombarded 
with  images  and  sound  via  today’s  media;  I  think  this  results  in  a  brief  bafflement  over 
the  formative  years,  which  is  sorted  out  as  one  matures. 

In  phase  three,  consistency  is  once  again  prevalent,  with  the  same  colors  as  phase 
one!  The  only  major  variations  deal  with  individual  in  their  50s.  Cuteitta  and  Haggerty 
(1987)  speculate  that  it  may  be  related  to  the  mid-life  “crisis”  period  of  development. 

After  analyzing  the  data  of  this  study,  Cuteitta  and  Haggerty  (1987)  were  faced 
with  the  following  questions: 

(1)  If  color/music  associations  are  a  conditioned  response,  why  isn’t  there  a 
gradual  rise  in  consistency? 

(2)  If  conditioning  is  from  media,  how  does  one  explain  the  consistency  of 
responses  from  older  subjects  who  were  not  exposed  to  color  media? 

Researchers  feel  that  the  results  from  this  and  other  studies  points  to  a  theory  that 
color/music  association  is  the  result  of  some  sort  of  sensory  processing  of  music,  and  that 
this  processing  may  be  related  to  emotional  responses  (Cuteitta  and  Haggerty,  1987). 
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Manually  Generated  Visual  Music 

Typically,  visual  music  involves  a  complex  union  of  images  and  sound  to  create  a 
experience  that  can  be  more  expressive  than  either  of  the  two  mediums  alone  (Pocock- 
Williams,  1992).  This  visual  music  ranges  from  the  depiction  of  a  story  to  the  abstract 
display  of  colors  and  images. 

Even  if  the  visuals  do  emphasize  or  complement  the  music,  more  than  likely,  they 
only  deal  with  a  subset  of  the  music’s  properties.  Keeping  that  in  mind,  I  still  find  it 
useful  to  examine  how  different  elements  of  music  are  dealt  with  in  different 
applications. 

Predetermined  Animations 

Even  though  the  visual  music  is  generated  manually,  computers  can  still  be 
involved.  Pocock-Williams  (1992)  used  computers  to  generate  the  graphics  for  several 
pieces  of  music.  The  process  she  describes  is  as  follow:  (1)  start  with  a  pre-existing  piece 
of  music,  (2)  listen  to  music  many  times  until  she  had  a  sense  of  the  overall  quality  of 
matching  images,  (3)  write  a  computer  program  to  animate  the  work. 

Below  are  descriptions  of  two  pieces  of  music  and  her  corresponding  animations: 

(1)  Homage  to  Kandinsky:  Improvisation  (Williams,  1989).  Wassily  Kandinsky 
is  an  artist  to  whom  Pocock-Williams  credits  her  interest  in  visual  music.  The  images  for 
this  piece  were  taken  from  one  of  Kandinsky’s  paintings:  circles,  rectangles,  and 
triangles.  The  music  begins  subdued,  the  corresponding  shapes  are  mostly  gray  in  color, 
with  the  shapes  emerging  out  of  a  black  background.  The  second  half  of  the  music 
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becomes  more  dramatic.  In  response  the  shapes  begin  to  pulsate.  There  is  no  mention  as 
to  whether  the  colors  change  with  the  mood  of  the  music. 

(2)  Occam’s  Razor  (Williams,  1989).  The  music  for  this  piece  is  described  as 
tense,  fast,  and  abstract.  The  predominant  imagery  consist  of  a  series  of  parallel  lines. 
The  movement  of  the  lines  is  at  times  erratic  to  give  them  a  ‘nervous’  quality.  The 
animation  is  laid  over  recorded  live-action  video  (the  contents  of  the  video  are  not 
mentioned).  Pocock- Williams  describes  the  result  as  fast-paced  and  discordant  with  an 
unresolved  ending.  Again,  there  is  no  mention  of  a  particular  use  of  colors. 

The  above  animations  are  interpretations  of  how  music  made  the  author  feel 
about  the  music.  Based  on  the  descriptions,  I  don’t  believe  that  there  was  any  attempt  to 
use  colors  to  symbolize  the  emotions  portrayed  by  the  music.  Movement  and  shape  were 
the  major  forms  of  representation.  Movement  was  used  to  depict  the  tempo,  while  the 
shapes  were  what  the  author  thought  would  best  depict  the  music  as  a  whole.  There  was 
no  reference  to  the  intensity  of  either  musical  piece. 

Fantasia 

Walt  Disney’s  full-length  movie.  Fantasia,  contains  a  series  of  classical  music 
pieces  that  are  accompanied  by  animation.  The  animations  are  interpretations  of  the 
music  as  seen  by  artist,  not  trained  musicians.  The  narrator  mentions  that  there  are  three 
types  of  music  depicted  in  the  movie:  (1)  music  that  tells  a  definite  story;  (2)  music  that 
does  not  tell  a  story,  but  does  produce  definite  images;  and  (3)  music  that  exist  for  its 
own  sake.  I  believe  the  typing  of  music  has  a  lot  to  do  with  the  title  given  to  the  piece. 
The  only  music  classified  as  ‘music  that  exist  for  its  own  sake’  was  Toccata  and  Fugue  in 
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D  minor  by  Johann  Sebastian  Bach,  its  title  is  more  a  description  of  the  music.  Likewise, 
since  the  title  is  not  explicit  neither  is  the  animation.  The  other  animations  in  the  movie 
were  very  explicit  and  didn’t  leave  much  to  the  imagination.  For  this  reason  I  focused  on 
Toccata  and  Fugue. 

Before  the  animation  begins,  silhouetted  images  of  the  orchestra  are  displayed 
playing  their  parts.  The  background  was  lit  by  different  colors:  red,  yellow,  green,  and 
blue.  The  majority  of  the  time  the  color  used  seemed  to  be  dependent  on  how  much 
energy  (volume  or  intensity)  was  coming  from  the  focused  instruments.  For  example, 
blue  was  used  for  the  harps  (passive),  while  red  was  used  for  the  drums  (powerful),  green 
and  yellow  were  used  in  the  same  manner,  respectively.  I  did  not  notice  a  relationship 
between  color  and  notes.  At  times  when  the  energy  of  the  music  went  from  low  to  high 
the  color  shifted  from  blue  to  red. 

When  the  animation  began,  the  first  thing  I  noticed  was  that  some  of  the  images 
were  symbolic  of  the  motions  that  a  conductor  or  musician  might  make.  For  example, 
there  were  multiple  bows  at  about  25  to  45  degree  angles  moving  up  and  down  to  the 
music.  The  emphasis  was  no  longer  on  the  colors,  but  on  having  the  abstract  images 
represent  parts  of  the  music.  Lines,  circles  and  stars  danced  across  the  screen  or  moved 
across  a  imaginary  terrain.  Ascending,  descending,  and  alternating  notes  were 
represented  by  the  images  doing  the  same  with  their  movements,  mapping  the  tempo  at 
the  same  time. 

The  animations  picked  the  most  obvious  elements  of  the  music,  melody  and 
tempo,  which  gave  the  overall  feel  of  the  music.  Watching  the  images  without  the  music 
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did  not  produce  the  same  effects.  This  is  not  surprising,  since  I  believe  the  goal  of  the 
animation  was  to  enhance,  not  represent,  the  music. 

Automatically  Generated  Visual  Music 

From  my  research,  I  have  found  very  few  attempts  to  automatically  generate 
visual  music.  When  I  did  find  them,  the  goal  of  the  software  was  to  accompany  and 
enhance  the  aural  experience  of  music  with  images  and  animation,  not  to  provide  an 
additional  medium.  Given  the  fact  that  the  visuals  are  meant  to  heighten  the  experience, 
they  only  deal  with  a  subset  of  the  music’s  properties,  most  commonly  pitch  and 
duration.  As  with  manually  generated  visual  music,  the  work  done  in  this  area  provides 
me  with  an  excellent  starting  point  and  frame  of  reference. 

Automatically  Generated  Animations 

Pocock-Williams  (1992)  also  did  some  work  generating  visual  images  from  music 
automatically.  She  developed  a  ‘visual’  language  consisting  of  basic  shapes,  colors,  and 
linear  movement.  The  computer  controls  time-sensitive  operations  such  as  scaling, 
translating,  pulsating,  and  replicating.  Attention  is  also  paid  to  how  objects  enter,  exit, 
and  cross  paths  in  the  viewing  area. 

The  software  used  in  this  project  was  SoundScape  Pro  MIDI  Studio  from 
Mimetics  Corporation.  SoundScape  stored  the  input  music-data  as  a  MIDI  file.  The 
author  wrote  programs  for  extracting  and  analyzing  data  from  the  MIDI  file,  and  creating 


the  animation. 
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She  used  these  four  major  steps  for  sound-to-music  integration:  (1)  music  input 
via  SoundScape  Pro  MIDI  Studio;  (2)  music  analysis  (3)  decision  making  according  to 
preset  rules,  and  (4)  animation  generation. 

Once  the  music  is  stored  into  the  MIDI  file  the  required  data,  pitch  and  duration, 
are  extracted  and  stored  into  a  text  file  which  is  then  converted  to  integer  format.  The 
data  is  then  read  in  at  10  second  intervals  for  analysis.  The  program  searches  for  four 
different  patterns.  If  the  pitches  are  all  increasing,  decreasing,  alternating  or  flat  they  are 
replaced  by  UP,  DOWN,  ALT,  and  FLAT  terminals,  respectively. 

Pocock-Williams  (1992)  provides  the  following  example: 

Assuming  the  order  of  pitches  from  high  to  low  is  as  follows: 
a  b  c  d  e  f, 

a  10-sec  phrase  of  notes  that  looks  like 

acac  cdf  fdc  acac  cdf  fdc  acac  cdf  fdc 
would  be  replaced  by  a  simplified  pitch-time  graph  that  looks  like 
ALT  UP  DOWN  ALT  UP  DOWN  ALT  UP  DOWN 

She  further  mentions  that  since  ALT  DOWN  UP  occurs  three  times  it  is  a 
‘predominant  pattern’  and  the  whole  thing  will  be  reduced  to  one  ALT  DOWN  UP. 

It  can  be  seen  from  this  example  that  the  relationship  from  music  to  animation 
will  not  be  one-to-one.  In  the  course  of  this  ten  second  interval  the  music  has  phased 
through  ALT  DOWN  UP  three  times,  but  whatever  actions  the  animation  will  take  on 
this  occurrence  will  happen  only  once.  This  should  give  the  general  feel  of  the  music, 
but  I  think  it  can  be  argued  that  something  will  be  missing. 

After  each  10  second  interval  is  analyzed  the  program  selects,  at  random,  one  of 
at  least  five  animation  terminals  that  has  been  pre-classified  as  being  a  visual  expression 
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of  the  selected  music  terminal.  For  the  above  example,  the  selected  animation  terminal 
would  exhibit  movement  that  corresponds  to  ALT  UP  DOWN.  Figure  2.4  displays 
examples  of  three  of  the  possible  five  animation  terminals  for  ALT  UP  DOWN.  Each 
animation  sequence  runs  about  10  seconds.  This  process  continues  in  10  second  intervals 
until  the  music  is  complete.  No  mention  was  made  as  to  whether  these  animations 
perform  their  cycles  once  or  many  times  in  the  10  second  interval. 


Figure  2.4  Animation  Terminals 
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Again,  it  is  easy  to  see  that  while  this  may  represent  the  overall  feeling  in  music, 
it  is  quite  possible  to  miss  out  on  the  many  of  the  intricacies  of  the  music.  It  is  also 
worth  repeating  that  this  animation  is  not  meant  to  stand  alone,  the  music  will  be  played 
simultaneously.  It’s  main  goal  is  to  enhance  the  musical  experience. 

Screen  Dance 

ScreenDance  version.  1.0  J  from  Madeira  Software,  Inc.  is  a  32-bit  CD  player  that 
comes  with  sound  activated  animations.  Each  animation  displays  real-time  imagery  in 
response  to  whatever  music  is  in  the  computer’s  CD  player. 
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ScreenDance  accomplishes  this  by  using  Fast  Fourier  Transform  (FFT)  to  analyze 
the  audio  signal  coming  from  a  CD  player  in  to  a  tabulation  of  frequencies  (pitches). 
From  there  it  gives  instructions  to  the  selected  animation.  The  exact  process  they  used  to 
go  from  frequencies  to  animation  was  not  available. 

The  downloadable  version  of  this  software  (http://www.madeira.com/dance.htm) 
contains  the  following  animations:  3D  spectrograph  that  plots  the  magnitude  of  the 
frequencies  over  time,  lower  notes  on  the  left  higher  on  the  right;  a  stick  figure  dancer 
(see  Figure  2.5);  fractals,  computer  generated  art  that  pulsates  to  the  sound; 
kaleidoscopes  that  moves  to  the  beat  of  the  music;  a  pile  of  marbles;  and  actual 
soundwaves. 

ScreenDance  makes  no  explicit  attempt  to  interpret  the  mood  of  the  music.  The 
animations  are  dictated  by  the  rate  of  change  in  the  tempo  and  pitch  of  the  music. 

Figure  2.5  ScreenDance  Dancer 


I  spoke  to  Dave  Young  of  Madeira  Software,  Inc,  he  said  what  they  are  trying  to 
do  with  this  software  is  to  enable  people  to  enjoy  music  to  a  fuller  extent.  Currently  they 
are  working  on  a  native  Windows  95/NT  version  of  ScreenDance,  with  plans  on  releasing- 
a  ScreenDance  SDK  (software  development  kit)  sometime  in  the  near  future,  which  will 
include  all  the  libraries  and  interfaces  you  would  need  to  create  your  own  animations  for 
the  Windows  version  of  ScreenDance.  It  will  be  designed  in  such  a  way  to  be  relatively 
easy  for  3rd  parties  to  create  new  animation  modules.  New  animation  modules  will  be 
programmed  in  C/C++.  The  animation  module  will  have  has  access  to  the  raw  digital 


32 


audio  data,  and  can  make  use  of  included  library  functions  for  extracting  frequency, 
intensity,  and  duration  information. 


CHAPTER  3 


METHOD 

The  goal  of  the  completed  software  is  to  capture  the  emotions/feelings  of  music 
and  translate  them  to  an  animated  display  using  color,  shape,  and  motion. 

Instrument 

The  instrument  which  I  will  focus  will  be  the  solo  violin.  Referring  to  Figure  2.1, 
one  can  see  that  the  violin  has  no  trouble  reaching  frequencies  on  either  side  of  the 
standard  A4  (440cps).  Given  this  fact,  I  will  be  able  to  use  the  predefined  musical 
characteristics  for  producing  various  emotional  expressions  (Table  2.1) 

The  violin  seems  to  be  one  of  the  more  popular  instruments.  In  most  books  that 
I’ve  come  across,  which  describe  musical  instruments,  stringed  instruments  usually  come 
first  and  violins  are  always  the  first  in  the  string  category.  Culver  (1956)  notes  that  the 
violin  is  the  basis  of  orchestral  instrumentation  and  in  the  hands  of  a  skilled  musician, 
has  an  almost  unlimited  capability  of  conveying  emotional  concepts. 

On  a  personal  note,  I  am  very  partial  to  violin  music.  I  find  the  violin  able  to 


portray  a  wide  range  of  emotions. 
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Music 

It  was  difficult  to  find  a  wide  variety  of  violin  solos  on  CD.  There  is  a  vast  array 
of  beautiful  violin  music  available,  but  for  the  most  part  it  is  accompanied. 

Unfortunately,  the  violin  solos  that  I  have  found  seem  to  concentrate  more  on  technique, 
which  can  make  them  somewhat  difficult  to  enjoy. 

My  goal  was  to  find  two  pieces  of  music,  one  that  was  primarily  sad  and  another 
that  was  primarily  happy.  The  two  violin  solos  I  selected  are  listed  below: 

24  Caprices  for  Solo  Violin,  Op.  1,  no  4  in  C  Minor,  composed  by  Nicolo 
Paganini,  performed  by  Michael  Rabin.  This  selection  is  primarily  on  the  sad  side, 
however  the  tempo  increases  after  the  first  60  seconds,  therefore  I  only  used  the  first 
minute.  The  melody  is  very  disjunct  and  disturbed.  I  had  to  listen  to  it  several  times 
before  I  began  to  enjoy  it. 

Sonata  no.  3  in  C  Major,  Allegro  Assai,  composed  by  Johann  Sebastian  Bach, 
performed  by  Lara  St.  John.  This  selection  is  primarily  on  the  happy  side.  I  used  the  first 
85  seconds  of  this  piece.  I  enjoyed  this  selection  the  first  time  I  heard  it.  Its  melody  is 
playful  and  cheerful. 

I  decided  to  add  a  third  piece  of  music  that  does  not  fall  into  the  violin  solo 
category.  The  music  I  chose  is  the  Theme  From  JAWS  -  composed  by  John  Williams  - 
performed  by  the  Boston  Pops  Orchestra.  This  piece  of  music  has  a  definite  link  to 
emotions  for  many  people,  whether  they  have  seen  this  movie  or  not.  Overall,  I  would 
classify  this  music  is  threatening,  exciting  and  powerful.  Since  this  is  not  a  solo 
arrangement  I  didn’t  expect  optimal  results,  but  It  did  provide  an  interesting  display.  I 
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still  collected  valuable  information  from  the  reactions  of  those  who  watched  the 
animation. 

Music  to  Emotion  to  Animation 

This  process  will  consist  of  two  major  steps.  First,  the  software  needs  to  identify 
the  emotion  of  the  music.  Secondly,  the  identified  emotion  must  then  be  translated  into 
color,  shape,  and  texture.  To  identify  emotion  in  music  I  used  the  data  presented  in 
Table  2.1  (Musical  Characteristics  for  Producing  Various  Emotional  Expressions)  as  a 
starting  point.  Likewise,  I  used  the  data  in  Table  2.3  (Color  and  Emotion  Associations) 
as  a  starting  point  in  representing  the  emotions  visually. 

Identifying  Emotion  in  Music 

Music  can  express  a  wide  range  of  emotions,  but  I  focused  on  those  which  I  feel 
are  core  and  easiest  to  identify.  Bruner  (1990)  presented  musical  characteristics  for  six 
emotional  expressions:  serious,  sad,  serene,  happy,  exciting,  and  frightening.  I  focused 
on  four  of  these:  sad,  serene,  happy,  and  exciting.  I  feel  that  serious  and  frightening  are 
both  subsets  of  sad  and  serene,  which  is  easy  to  see  by  comparing  the  attributes  for  all 
four  (Table  2.1). 

Bruner  (1990)  used  six  musical  elements  to  describe  the  emotional  expressions: 
mode,  tempo,  pitch,  rhythm,  harmony,  and  volume.  Of  these  six,  I  only  used  three:  pitch, 
volume,  and  tempo  (duration).  Taking  a  close  look  at  Table  2.1  will  reveal  that  of  the 
different  emotions  listed,  all  have  a  different  combinations  of  the  three  attributes  that  I 
chose.  Harmony  is  not  a  factor  since  I’m  concentrating  on  one  instrument,  in  solo 
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performances.  The  rhythm  of  the  music  can  be  broken  down  into  volume  and  duration. 
The  mode  of  the  music  can  be  decomposed  into  frequency  ranges. 

The  emotions  I  chose  are  on  opposite  ends  of  the  spectrum;  the  sad  side,  which 
includes  serene  and  the  happy  side,  which  includes  exciting.  I  could  have  just  chose 
happy  and  sad,  but  serene  and  exciting  give  me  good  reference  points  from  which  to 
expand.  These  two  sides  reflect  different  energy  levels.  Low  energy  emotional 
expressions  like  sad,  serious,  and  serene  are  characterized  by  slower  movements,  which 
is  depicted  by  slow  tempo  (long  durations).  Conversely,  high  energy  emotional 
expressions,  such  as  happy,  angry,  and  joy  are  characterized  by  faster  movements,  which 
is  portrayed  by  a  faster  tempo  (short  durations).  So,  what  we  have  is  a  ‘tempo  divide’, 
low  energy  emotions  are  portrayed  by  slower  tempos  and  high  energy  emotions  are 
portrayed  by  faster  tempos. 

As  stated  above,  I  used  pitch,  volume,  and  tempo  to  categorize  emotional 
qualities  in  music.  I  took  the  descriptions  of  emotional  expression  for  sad,  serene,  happy 
and  exciting  from  Bruner  (1990)  and  expanded  from  there.  This  provided  me  with  four 
definitions  from  which  to  start,  but  left  many  other  options  unexplained.  Given  that  pitch 
and  volume  can  each  be  classified  as  (1)  low,  (2)  medium,  and  (3)  high  and  tempo  as  (1) 
slow  and  (2)  fast.  We  are  presented  with  18(3x3x2)  possible  combinations  of  the 
three  attributes.  Each  side  of  the  tempo  divide  includes  nine  combinations.  Those  nine 
combinations  are  broken  down  further  on  each  side  of  the  tempo  divide. 
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Table  3. 1  Emotional  Expressions  Related  to  Musical  Elements  for  Violin 


Sad  Serene  o  ®  Happy  Exciting 

.  CL  T3 . . . , 


—  —  " 

"■  X  ™ 

X  -  - 

X  X  - 

+  -  - 

+  X  - 

Volume 

-  +  - 

Divide 

1 

+ 

X 

+  +  - 

pC  ©  © 

ptf  o 

i  I  t 

w  g  a 

•ts  5  g 

fc  |  s 

©  lZ 
>  H 

>  H 

lO  +  X  + 

X  +  + 

+  -  + 

+ 

1 

X 

;  +  +  + 

X  X  + 

i  Pitch 

-  -  + 

'  Divide 

+ 

X 

1 

-  +  + 

pC  2  © 

js  2  ° 

v  c  a. 
On  |  S 

s  s  §■ 
s  |  1 

>  H 

>  H 

Source:  Data  for  top  line  of  table  extracted  from  Gordon  Bruner,  “Music, 
Mood,  and  Marketing.”  Journal  of  Marketing  54,  no.  4  (1990):  100. 

Note:  Symbol  meanings  are  as  follows:  “  =  slow,  low,  soft;  X  =  medium;  + 
=  fast,  high,  loud.  Actual  meaning  depends  on  which  musical  attribute 
symbol  is  attached.  Expansion  from  the  first  line  of  the  table  are  my  own 
observations.  The  order  of  all  combinations  is  pitch,  volume,  tempo. 


Table  3.1  list  all  possible  combinations  of  pitch,  volume,  and  tempo*.  As  can  be 
seen,  a  volume  divide  occurs  on  the  sad  side  and  a  pitch  divide  occurs  on  the  happy  side. 
The  expansion  of  the  combinations  beyond  those  given  by  Bruner(1990)  are  from  my 
own  observations  and  opinions.  I  spent  many  hours  listening  to  various  pieces  of  music 
to  determine  the  combinations  in  Table  3. 1 . 

On  the  sad  side  we  have  the  nine  PVTs  partitioned  into  three  categories  based  on 
volume.  Looking  at  the  three  sad  PVTs,  the  only  difference  between  them  is  the  pitch. 


*  PVT  will  be  used  for  future  reference  to  pitch,  volume,  and  tempo,  when  mentioned  together.  A 
combination  of  three  symbols  -,  x,  or  +  may  follow  (eg.,  PVT  -x-)  or  stand  alone  denoting  the  value  of  each 
respective  element.  See  the  note  following  Table  3.1  for  definitions  of  the  symbols. 
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In  violin  music,  I  found  that  sad  music  can  occur  at  all  pitch  levels,  this  emotion  seems 
most  dependent  upon  tempo  and  volume.  The  level  of  pitch  can  determine  the  intensity 
of  the  sadness.  While  listening  to  music,  I  found  that  the  higher  the  pitch  the  more 
physically  tense  I  became,  I  can  actually  feel  my  muscles  tightening.  The  opposite  would 
occur  with  lower  pitches,  it  seemed  easier  to  relax. 

The  three  PVTs  directly  below  serene  are  similar  to  sad  except  for  volume.  The 
differences  among  the  three  are  only  in  pitch,  which  determines  the  level  of  sereness. 

The  last  three  PVTs  under  serene  are  not  explicitly  emotional  expressions  of  serene. 

They  are  combinations  which  are  not  necessarily  sad  or  serene,  but  are  on  the  same  side 
of  the  tempo  divide  due  to  their  slow  tempo.  The  increased  volume  seems  to  cause  an 
enlightenment  of  the  emotion,  possibly  creating  a  bridge  to  emotions  on  the  happy  side.  I 
do  not  personally  know  of  any  music  that  sustains  high  volumes  at  a  slow  tempo,  instead 
music  that  fits  this  category  may  raise  its  volume  temporarily  for  accent  or  impact. 

On  the  happy  side,  pitch  is  the  divider.  I  found  that  happy  music  can  occur  at 
different  volumes.  The  lower  volumes  seems  more  playful  and  small.  The  louder  music 
sounds  more  energetic  and  large.  The  same  logic  holds  true  for  the  first  three 
combinations  under  exciting.  The  last  three  PVTs  under  exciting  are  similar  to  the  last 
three  under  serene  in  that  they  are  not  necessarily  happy  or  exciting,  but  they  exist  on  the 
same  side  of  the  tempo  divide  due  to  their  fast  tempo.  Unlike  the  sad  side,  it  is  a  bit 
easier  to  see  what  emotions  these  PVTs  may  portray.  In  general,  music  with  a  fast  tempo 
and  low  pitch  can  sound  powerful. 
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So  far,  only  two  emotions  have  more  or  less  precisely  defined;  happy  and  sad,  and 
even  they  have  three  PVTs  each.  The  emotions  that  the  other  PVTs  portray  will  be 
defined  automatically  via  the  color  and  animation  (shape,  size,  and  texture).  I  believe 
that  if  the  PVT  is  correctly  classified,  the  visualization  of  the  selected  attributes  will 
mirror  the  correct  emotion.  This  will  be  explained  in  more  detail  later. 

Portraying  Emotion  with  Colors 

After  identifying  the  emotion  of  the  music  the  next  step  is  to  use  color  to  visually 
represent  the  emotion.  Table  3.2  summarizes  the  conversion.  The  colors  used  are  those 
of  the  standard  color  wheel  and  the  color-emotion  references  are  derived  from  Birren 
(1961)  and  Sharpe  (1974)  studies.  Sad  and  happy  have  direct  links  to  specific  colors 
while  serene  and  exciting  point  to  two  different  colors. 

Recalling  Table  2.3,  we  see  that  blue  and  green  share  some  emotion 
associations,  but  they  do  differ  slightly.  Green  is  quieting,  refreshing  and  peaceful,  while 
blue  is  subduing,  melancholy,  and  sober.  Blue  seems  to  represent  deeper  emotions.  For 
the  musical  element  combinations  in  the  serene  category,  those  with  lower  pitches  are 
related  to  blue,  while  the  green  is  used  for  the  higher  pitches. 

Orange  and  red  are  also  similar,  but  different.  Red  is  passionate,  exciting,  and 
active,  while  orange  is  jovial,  lively,  energetic,  and  forceful.  Overall,  red  seems  more 
serious,  it  is  assigned  those  combinations  under  exciting  that  have  a  lower  pitch.  Since 
orange  is  more  lively  and  closer  to  happy  it  is  assigned  those  combinations  under  exciting 
that  have  a  higher  pitch. 
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Table  3.2  Emotion  of  Music  to  Color 
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Sources:  Data  from  Gordon  Bruner,  “Music,  Mood,  and  Marketing.” 
Journal  of  Marketing  54,  no.  4  (1990):  100;  Faber  Birren.  Color 
Psychology  and  Color  Therapy  -  A  Factual  Study  of  Color  on  Human 
Life.  New  York:  University  Books,  Inc.,  1961.;  Deborah  Sharpe.  The 
Psychology  of  Color  and  Design.  Chicago:  Nelson-Hall  Company, 
1974. 


Object  Shape  and  Movement 

Since  I  will  be  focusing  on  solo  violin  there  is  a  single  object  on  the  screen 
depicting  the  emotions  it  portrays.  The  shape  of  the  object  will  depend  on  several  factors 
involving  the  emotion,  pitch,  and  volume. 

The  default  shape  will  be  circular.  Higher  pitches  will  cause  the  shape  to 
constrict  on  the  horizontal  plane,  while  low  pitches  will  cause  the  shape  to  widen 
horizontally.  As  mention  before,  for  me,  higher  pitches  are  more  tense  and  compact  than 
lower  pitches.  Also,  the  higher  the  frequency  of  a  pitch  the  shorter  its  wavelength, 
conversely,  lower  frequencies  have  longer  wavelengths.  Constricting  the  shape 
horizontally  for  high  frequencies  is  comparable  to  tensing,  while  widening  horizontally  is 
equivalent  to  a  loosening  of  its  containment.  Figure  3.1  displays  relative  examples  of  a 
circular  shape  compared  to  the  frequency  of  pitch. 
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Figure  3.1  Pitch  Shapes 


The  object’s  outer  texture  is  determined  by  comparing  consecutive  pitches. 
Conjunct  pitches  are  considered  sweet,  happy,  and  lyrical  while  disjunct  pitches  are 
dramatic,  energetic  and  agitated.  Conjunct  pitches  will  produce  a  smooth  skin  while 
disjunct  pitches  will  produce  a  more  jagged  skin.  Figure  3.2  shows  the  relative  example 
of  a  disjunct  texture.  Figure  3.1  can  also  be  considered  an  example  of  conjunct  texture. 
The  shapes  will  also  mirror  the  level  of  the  which  the  pitch  is  disjunct.  A  more  disjunct 
pitch  will  produce  a  shape  which  is  more  jagged. 


Figure  3.2  Disjunct  Pitch  Shapes 


The  objects  size  is  determined  by  the  current  volume.  Volume  is  related  to 
energy  level  and  depth.  On  a  two  dimensional  plane  the  depth  of  an  object  may  be 
represented  by  size.  The  closer  the  object  is  the  larger  it  appears.  Hence,  the  louder  the 
music  the  larger  the  object. 
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Figure  3.3  Volume  Size 


Putting  It  Together 

Each  PVT  is  a  sample  of  the  music’s  attributes  for  a  fraction  of  a  second,  a  very 
small  unit  of  the  music  as  a  whole.  As  each  PVT  is  animated  the  emotion  of  the  music 
will  be  revealed.  Although,  only  two  emotions  are  closely  defined,  a  wide  range  of 
emotions  can  be  identified  and  depicted  using  the  above  techniques  for  visual  portrayal. 
For  example,  the  feeling  of  surprise,  while  not  specifically  defined,  can  be  portrayed 
easily. 

In  music,  one  way  that  a  feeling  of  surprise  can  brought  on  is  by  a  sudden  change 
in  volume  level.  Let’s  look  at  two  consecutive  PVTs,  --+  and  -++,  which  denote  a  drastic 
change  in  volume  level  and  examine  the  corresponding  animation.  For  PVT  --+,  the 
object  would  be  wide,  small  and  red.  The  transition  to  PVT  -++,  would  call  for  the 
object  to  be  wide,  large,  and  red.  The  drastic  change  in  PVTs  would  cause  the  object  to 
explode  from  a  small  size  to  a  large  size  in  one  frame  of  animation,  therefore  causing  a 
surprise,  something  unexpected.  The  object  in  Figure  3.3  displays  an  extreme  size 
difference  that  may  be  applied  to  this  example. 
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Another  example  is  the  previously  mentioned  emotional  expression  of  fright.  I 
said  that  I  considered  fright  a  subset  of  the  sad  side.  The  characteristics  of  fright  would 
give  it  the  PVTs  of  ---,  -x-  and  -+-,  because  of  the  varied  volume.  The  PVTs  would 
produce  the  following  objects,  respectively:  wide,  small,  and  purple;  wide,  medium  size, 
and  blue;  wide,  large,  and  green.  The  animation  would  show  the  object  pulsing  through 
the  different  colors  and  sizes.  The  texture  of  its  skin  would  likely  be  ragged,  since  it’s 
likely  to  be  disjunct. 

Other  feelings  of  music  like  ascension  (rise  in  pitch  over  time),  crescendo 
(gradual  increase  in  volume),  and  downward  pull  (decrease  in  pitch  over  time)  to  name  a 
few  will  all  be  mirrored  visually  by  the  form  the  object  assumes  over  time. 

Software  -  See  Symphony 

The  purpose  of  this  software  is  to  demonstrate  my  hypothesis  that  the  emotions  of 
music  can  be  automatically  identified  and  displayed  visually.  The  software  is  named  See 
Symphony,  it’s  named  for  its  ability  to  allow  those  to  see  what  they  cannot  hear.  The 
source  code  will  not  be  listed  for  copyright  reasons.  The  SDK  provided  by  Madeira 
software  is  a  Beta  version,  the  release  version  is  due  out  in  early  1998. 

Design 

See  Symphony  accomplishes  the  following  steps:  (1)  read  in  the  music  data,  (2) 
extract  PVT  from  music,  (3)  determine  emotion  of  music,  (4)  generate  and  display  frame 
of  animation,  and  (5)  repeat  steps  1  though  4  until  music  is  over.  The  following  data 
flow  diagram  (Figure  3.4)  depicts  how  the  data  will  flow  through  the  software. 
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Figure  3.4  See  Symphony  Data  Flow  -  Level  1 


Pitch 


I  used  the  ScreenDance  SDK  (Beta)  developed  by  Madeira  Software,  Inc.  as  my 
platform.  ScreenDance  includes  a  musical  CD  interface  and  provides  procedures 
(C/C++)  for  extracting  data  from  raw  music.  The  crux  of  my  programming  work 
consisted  of  writing  procedures  (C/C++)  to  analyze  the  data  and  generate  the  animation. 
Figure  3.5  displays  a  decomposition  of  the  “Extract  Data”  bubble.  All  processes  in 
Figure  3.5,  except  “Select  Significant  Pitch”  and  “Determine  Note  Duration”  are 
provided  by  the  ScreenDance  SDK. 
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Figure  3.5  Decomposition  of  "Extract  Data"  Process  -  Level  2 

Pitch 


Once  the  data  is  extracted  it  is  analyzed  for  emotional  properties,  pitch,  and 
volume.  All  of  these  factors  are  used  to  generate  the  animation  instructions.  The 
“Analyze  Data”  bubble  is  decomposed  in  Figure  3.6. 
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Figure  3.6  Decomposition  of  "Analyze  Data"  Process  -  Level  2 


Challenges 

There  were  two  main  problems  that  I  had  to  address  to  make  See  Symphony 
mirror  the  music  as  closely  as  possible.  First  and  foremost  was  the  issue  of  frequencies. 
The  function  provided  by  Madeira  Software  Inc.  that  extracted  the  frequency  information 
from  the  music  stores  the  data  in  a  linear  form.  As  one  can  see  from  Table  2.2 
(Frequencies  of  Notes)  the  frequencies  in  the  musical  scale  are  not  linear.  In  order  to 
properly  classify  the  frequencies  I  created  bins  that  represented  each  note  on  the  scale. 
The  frequency  information  from  the  frequency  function  was  then  sorted  into  the  bins. 
From  there  I  was  able  to  more  accurately  determine  which  note  was  peaking  at  a  given 
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moment.  This  also  becomes  important  in  determining  whether  a  melody  (consecutive 
frequencies)  is  conjunct  or  disjunct. 

For  See  Symphony  to  decide  whether  consecutive  frequencies  were  conjunct  or 
disjunct  it  simply  compares  the  bin  numbers  of  the  frequencies.  If  the  bin  numbers  are 
more  than  a  specified  distance  away  from  each  other  then  they  are  labeled  disjunct. 

The  second  major  issue  was  that  of  tempo.  The  function  that  Madeira  Software 
Inc.  would  have  provided  looked  at  peaks  in  the  volume  to  determine  tempo.  While  this 
approach  would  work  well  for  Pop  music  or  any  music  with  a  drum  beat  or  consistent 
accents  I  did  not  think  that  it  work  well  with  classical  music.  Instead  I  wrote  a  function 
that  examines  frequency  duration  as  described  in  Chapter  2.  The  function  measures  the 
length  of  time  that  a  note  or  silence  is  held  and  then  makes  a  determination  over  time  if 
the  tempo  is  fast  or  slow. 

Customizing 

In  order  to  make  the  animations  work  best  for  each  individual  I  allowed  for  the 
customization  of  certain  parameters,  such  as  color  and  emotion  associations.  Other 
parameters  are  changeable,  but  these  are  mostly  for  experimentation  to  prevent  the  need 
to  recompile  for  every  adjustment.  The  control  panel  for  the  various  changeable 
parameters  is  shown  in  Figure  3.7. 
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Figure  3.7  Customize  Control  Panel 
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Cancel 
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According  to  the  subject’s  preferences  the  colors  for  each  emotion  category  could 
be  changed  to  any  one  of  the  six  of  the  standard  color  wheel  (red,  orange,  yellow,  green, 
blue,  purple).  The  emotions  associated  with  each  category  are  listed  in  Table  3.3. 


Table  3.3  Default  Color  to  Emotion  Association 


Emotional  Category 

Default  Color 

Emotions/Feelings  in  Category 

SAD 

Purple 

Sad,  Mournful,  Royal 

SERENE  1 

Blue 

Subduing,  Sober,  Calm 

SERENE  2 

Green 

Refreshing,  Peaceful 

HAPPY 

Yellow 

Happy,  Cheerful,  Fun 

EXCITE  1 

Orange 

Lively,  Energetic,  Jovial 

EXCITE  2 

Red 

Exciting,  Passionate,  Stimulating 
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The  other  options  on  the  control  panel  in  Figure  3.7  are  for  experimental  and 
adjustment  purposes,  definitions  for  these  parameters  are  listed  in  Table  3.4. 


Table  3.4  Customization  Parameters 


Parameter 

Function 

Volume  Slew 

Sets  the  sensitivity  to  volume  change.  Ranges  from  1  to  100.  A  higher 
number  causes  the  software  to  be  more  sensitive  to  fluctuations  in 
volume  level.  A  low  setting  would  require  more  of  a  change  in  the  actual 
volume  before  the  software  senses  a  difference  in  volume  level.  Default 
is  100. 

Conjunct  Tolerance 

Sets  how  many  notes  consecutive  frequencies  must  be  apart  before  a 
melody  is  considered  disjunct.  Ranges  from  0  to  5.  Default  is  3. 

Note  Accuracy 

Sets  accuracy  of  frequency  function.  Setting  are  LOW,  MED,  or  HIGH. 

A  higher  setting  means  more  samples  will  be  taken  at  each  instance. 

More  samples  means  higher  accuracy,  but  it  also  requires  more  processor 
time.  Default  is  MED. 

Medium  Pitch  Low 

Sets  the  lower  boundary  for  medium  pitch.  The  value  is  a  bin  number. 
Ranges  from  1  to  29.  Default  is  8. 

Medium  Pitch  High 

Sets  the  upper  boundary  for  medium  pitch.  The  value  is  a  bin  number. 
Ranges  from  1  to  29.  Default  is  15. 

Slow  Tempo  Duration 

Sets  the  time  for  a  long  duration  in  1/10  seconds.  Ranges  from  1  to  10. 
Any  duration  greater  than  setting  is  considered  long  duration.  Default  is 

2. 

Tempo  Tolerance 

Sets  the  tolerance  (out  of  10)  for  long  durations  (slow  tempo)  Ranges 
from  1  to  10.  If  the  specified  number  of  long  duration  occurs  out  of  10 
samples  then  the  tempo  is  considered  slow.  Default  is  2. 

Running  See  Symphony 

See  Symphony  works  much  better  than  expected.  It  is  able  to  pick  out  very 
accurate  information  about  the  music  at  each  instance.  The  one  drawback  is  that  it  needs 
a  relatively  fast  computer  to  work  properly.  The  best  output  is  achieved  when  the 
animation  is  between  20  to  30+  frames  per  second  (fps).  To  achieve  at  least  20fps  I  used 
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a  computer  (notebook)  with  a  1 50MHz  Pentium  Processor  with  MMX  technology.  To 
achieve  30+fps  I  used  a  computer  with  a  166MHz  Pentium  Processor  with  MMX 
technology.  The  processor  must  do  a  large  amount  of  computations  many  times  a 
second.  The  result  is  a  real-time  visual,  animated  display  of  the  music. 

The  majority  of  the  test  was  accomplished  with  the  150MHz  Pentium  computer. 

Testing 

See  Symphony  was  tested  in  two  phases.  The  first  phase  involved  5  individuals,  3 
of  whom  were  deaf.  Each  individual  watched  the  animations  separately.  Phase  I  was 
very  structured  and  its  purpose  was  to  record  individual  reactions  to  music  and 
animations  to  determine  if  See  Symphony  was  capable  of  visually  portraying  the 
emotions  of  music.  Phase  II  involved  a  group  of  10  people,  all  hearing.  The  members  of 
this  group  watched  the  animations  at  the  same  time.  The  purpose  of  this  phase  was 
simply  to  observe  group  reactions. 

Phase  I 

The  three  deaf  subjects  and  two  hearing  subjects  watched/listened  to  the 
animations/music  individually.  I  had  hoped  to  have  more  hearing  impaired  participants, 
but  two  canceled  due  to  illness. 

The  sample  size  for  this  phase  is  relatively  small,  but  I  don’t  believe  the  lack  of 
many  subjects  is  a  crucial  factor.  If  See  Symphony  works  for  a  even  one  of  the  hearing 
impaired  individual  then  I  consider  it  a  success. 
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Each  person  from  this  group  filled  out  one  of  two  questionnaires,  depending  on 
hearing  ability.  Samples  of  both  questionnaires  and  cover  sheets  can  be  found  in  the 
appendix.  The  questionnaire  is  divided  into  three  sections.  Section  I  ask  a  few  person 
questions.  Section  II  asks  the  subject  to  match  colors  with  emotional  categories.  The 
subjects  were  given  samples  of  the  six  primary  colors  to  aid  in  the  color-emotion 
association.  When  the  subjects  completed  Section  II,  I  adjusted  the  color-emotion 
association  parameters  in  the  software  (Figure  3.7)  to  match  their  selections  prior  to 
running  the  animation.  Section  III  ask  the  subject  to  listen/watch  three  music  example 
and  record  their  reactions  to  the  music  and  animations. 

The  main  difference  in  the  two  questionnaires  was  how  they  record  responses  to 
the  music  and  animation.  The  hearing  impaired  subjects  were  asked  to  describe  the 
emotions  or  feelings  that  they  thought  the  animation  displayed.  The  hearing  subjects 
were  asked  to  describe  the  emotions  or  feelings  that  they  thought  the  music  portrayed. 
Subsequently,  they  were  asked  how  well  they  thought  the  animation  corresponded  to  the 
emotions  and  feelings  of  the  music.. 

My  reasoning  for  including  the  hearing  subjects  was  to  achieve  a  baseline  for  the 
emotions  and  feelings  of  the  selected  pieces,  beside  my  own  interpretations.  It  also  was  a 
way  to  find  out  if  the  animations  corresponded  well  to  the  music  for  people  with  hearing. 
I  felt  if  the  animations  corresponded  well  for  the  hearing  subjects  then  I  was  on  the  right 
track. 

I  ran  See  Symphony  from  a  notebook  computer.  The  primary  display  was  a  15 
inch  external  monitor.  The  notebooks  screen  acted  as  the  secondary  display.  All 
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subjects  could  only  see  the  primary  display.  Figure  3.8  shows  the  relative  positioning  of 
myself  and  the  test  subjects.  An  interpreter  was  present  during  the  sessions  with  the  deaf 
subjects. 


Figure  3.8  Individual  Presentation  Layout 

Secondary  Display 


Speaker 


Test  Subject 


Interpreter 


There  were  speakers  on  either  side  of  the  monitor,  each  of  these  was  set  at  high 
volume.  The  deaf  subjects  were  encourage  to  make  contact  with  the  desk  or  speakers  to 
pick  up  any  vibrations  that  would  emanate  from  the  speakers.  All  three  hearing  impaired 
subjects  did  so  during  the  course  of  the  testing. 

During  the  course  of  the  test  I  did  not  look  at  any  of  the  music/animation 
responses  nor  did  I  influence  the  subjects  in  any  way. 

Phase  II 

The  ten  participants  in  this  phase  were  all  hearing.  All  ten  are  either  employed  or 
associated  with  a  business  called  Meeting  the  Challenge,  Inc.,  located  in  Colorado 
Springs,  Colorado.  Meeting  the  Challenge  produces  software  and  hardware  products  to 
aid  in  and  improve  the  lives  of  those  with  disabilities.  Given  the  nature  of  their  business. 


I  felt  that  these  individuals  were  well  suited  to  share  their  thoughts  and  ideas  on  See 
Symphony.  More  information  about  this  business  is  available  on  the  Internet  at 
http://www.mtc-inc.com. 

The  individuals  in  this  group  did  not  fill  out  questionnaires,  but  some  did 
document  their  reactions.  These  reactions  can  be  found  in  the  Appendix  D. 

Figure  3.9  represents  the  general  position  the  two  displays  and  the  participants  in 
the  room  during  the  group  presentation. 

Figure  3.9  Group  Presentation  Layout 


All  in  the  room  had  a  good  view  of  the  monitor,  also  some  could  see  the  smaller 
secondary  display  on  the  notebook  computer.  Due  to  the  orientation  of  the  display,  size 
of  the  room,  and  the  number  of  people,  some  had  a  better  view  than  others. 


CHAPTER  4 


RESULTS 


Questionnaire  Results 

The  results  of  the  questionnaires  are  divided  into  two  categories:  hearing  impaired 
and  hearing.  Each  subject  read  cover  letters  which  gave  a  description  of  the  project  and 
instructions  regarding  the  music  animation.  These  cover  sheets  can  be  found  in  the 
appendix  along  with  the  questionnaires. 

Hearing  Impaired  Results 

The  following  tables  list  the  questionnaire  responses  for  the  hearing  impaired 
subjects.  Table  4. 1  corresponds  to  the  first  section  of  the  questionnaire  which  gathers 
some  personal  data.  Table  4.2  covers  Section  II,  which  ask  the  subjects  to  match  colors 
with  emotional  categories.  Table  4.3  is  the  subjects’  response  to  the  each  of  the  three 


animations  described  in  Section  III. 
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Table  4. 1  Responses  to  Questionnaire  (Hearing  Impaired) 


Questionnaire  Questions 

Subject  1 

Subject  2 

Subject  3 

Age? 

62 

15 

15 

Level  of  Hearing 
Impairment? 

Full 

Partial.  Can  hear  close 
noises  in  right  ear  and 
far  noises  in  left  ear 

Partial.  Can  hear  loud 
noises. 

Hearing  Impaired  from 
birth? 

No.  Lost  hearing  at  age 
13. 

No.  Lost  hearing  at  age 

6. 

No.  Lost  hearing  at  18 
months 

Use  any  devices  to  aid 
hearing? 

No. 

Yes.  Colloquial  implant 

No. 

Any  Experience  with 
music? 

Yes.  Listening  and 
singing  as  a  child.  No 
preferences 

Yes.  Listening.  No 
preferences 

No. 

Ever  been  involved  in 
other  studies  involving 
music  and  hearing 
impairment? 

No. 

No. 

No. 

Table  4.2  Color  Selection  (Hearing  Impaired) 


Emotion  Category 

Default 

Subject  1 

Subject  2 

Subject  3 

SAD 

Purple 

Purple 

Orange 

Red 

SERENE  1 

Blue 

Blue 

Red 

Yellow 

SERENE  2 

Green 

Green 

Yellow 

Purple 

HAPPY 

Yellow 

Red 

Blue 

Green 

EXCITE  1 

Orange 

Orange 

Green 

Blue 

EXCITE  2 

Red 

Yellow 

Purple 

Orange 

Note:  The  subjects  were  asked  to  select  a  color  that  they  felt  most  closely  portrays  the 
emotions/feelings  in  each  given  category.  They  drew  lines  from  each  color  to  an 
emotional  category.  A  swatch  of  each  color  was  provided  to  aid  in  the  visualization 
process. 
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Table  4.3  Responses  to  Music/Animation  (Hearing  Impaired) 


Music 

Subject  1 

Subject  2 

Subject  3 

24  Caprices  for  Solo 
Violin,  Op  1,  No  4  in  C 
Minor  -  Paganini 

Feels  subdued,  calming 
and  feels  good 

Good 

Felt  a  little  bit  like  loud 
(1  time) 

Sonata  No.  3  in  C 

Major,  Allegro  Assai  - 
Bach 

Emotional,  want  to  get 
up  and  dance.  Great! 

Exciting 

Could  hear  (loud  parts), 
loud  and  exciting 

Theme  from  JAWS  - 
John  Williams 

Feels  great  -  wild! !  Fast 
dancing  music  which 
makes  you  want  to  let 
go  and  dance  wildly! 

Exciting 

Preferred  lower  volume 
shapes.  Fun,  preferred 
this  one 

Notes:  The  subjects  were  asked  to  describe  the  emotions  or  feeling  the  animation  displayed. 


Hearing  Results 

The  following  tables  list  the  questionnaire  responses  of  the  hearing  subjects. 
Table  4.4  corresponds  to  the  first  section  of  the  questionnaire  which  gathers  some 
personal  data.  Table  4.5  covers  Section  II,  which  ask  the  subjects  to  match  colors  with 
emotional  categories.  Table  4.6  is  the  subjects’  response  to  the  each  of  the  three 


animations  described  in  Section  III. 
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Table  4.4  Responses  to  Questionnaire  (Hearing) 


Questionnaire  Questions 

Subject  4 

Subject  5 

Age? 

48 

37 

Any  Experience  with 
music? 

Yes.  I  listen  to  radio  music  at  work 
and  CD’s  at  home  on  stereo  and 
computer.  I  am  a  mood  listener: 
hard  busy  day  =  loud,  hard  rock  or 
metal;  easy  day  =  soft  rock  or  easy 
listening.  Periodically  listen  to 

Bach,  Wagner,  or  Tchaikovsky 

Yes.  Flute  player  since  grade  4, 
participated  in  marching  band  and 
orchestra  through  grade  12.  Like  a 
wide  variety  of  music:  blues, 
alternative,  “oldies”,  bluegrass,  and 
some  classical 

Ever  been  involved  in 
other  studies  involving 
music  and  hearing 
impairment? 

No. 

No. 

Table  4.5  Color  Selection  (Hearing) 

Emotion  Category 

Default 

Subject  4 

Subject  5 

SAD 

Purple 

Purple 

Purple 

SERENE  1 

Blue 

Blue 

Blue 

SERENE  2 

Green 

Green 

Green 

HAPPY 

Yellow 

Orange 

Yellow 

EXCITE  1 

Orange 

Yellow 

Orange 

EXCITE  2 

Red 

Red 

Red 

Note:  The  subjects  were  asked  to  select  a  color  that  they  felt  most  closely 
portrays  the  emotions/feelings  in  each  given  category.  They  drew  lines 
from  each  color  to  an  emotional  category.  A  swatch  of  each  color  was 
provided  to  aid  in  the  visualization  process. 
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Table  4.6  Responses  to  Music/Animation  (Hearing) 


Music 

Subject  4 

Subject  5 

24  Caprices  for  Solo 
Violin,  Op  1,  No  4  in  C 
Minor  -  Paganini 

(Music)  Piece  seemed  powerful  and 
yet  soothing  with  some  struggle 

(Animation)  The  themes  were  there, 
but  were  somewhat  confusing,  didn’t 
flow. 

(Music)  Soft,  subtle  feelings,  some 
sad,  but  immediately  relieved  -  not 
too  serious 

(Animation)  Great,  liked  the  size 
changes  -  large  to  small  -  really 
expressed  the  music.  Color  changes 
were  a  little  confusing  until  I  relaxed 
and  just  listened. 

Sonata  No.  3  in  C 

Major,  Allegro  Assai  - 
Bach 

(Music)  Piece  seemed  light  more 
active  and  felt  friendlier  and  happy 

(Animation)  Corresponded  well  and 
seemed  closer  to  the  theme,  more 
understandable. 

(Music)  Perky,  playfulness,  almost 
like  a  child  wanting  to  go  play, 
tugging  at  mother’s  hand,  who  holds 
him  back  with  a  little  restraint 

(Animation)  Good,  The  spikes 
showed  the  staccato  of  the  music, 
with  vibrant  colors  emphasizing  the 
happy  feelings. 

Theme  from  JAWS  - 
John  Williams 

(Music)  Experienced  a  building  of 
emotion  that  began  little  and  went  to 
powerful,  fear,  and  anxiety. 

(Animation)  Corresponded  well, 
wasn’t  confusing  because  the 
progression  was  evident 

(Music)  Deep,  dark,  and  growing  at 
the  beginning,  a  little  ominous,  but 
then  changing  to  something  very 
expressive,  a  little  lighter. 

(Animation)  Very  well  -  small  and 
dark,  but  growing,  changing  color 
and  sharpness  towards  the  end. 

Note:  The  subjects  were  asked  to  describe  the  emotion  or  feelings  that  they  thought  they  music  portrayed. 
Then  they  were  asked  how  they  thought  the  animation  corresponded  to  the  emotions  and  feelings  of  the 
music 


Discussion 

This  discussion  covers  data  from  the  questionnaires  of  both  the  deaf  and  hearing 
subjects.  Topics  discussed  are  the  color-emotion  associations,  responses  to  the 
animations,  and  general  observations. 
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Color-Emotion  Association 

The  color-emotion  association  for  most  subjects  was  very  close  to  the  default 
colors.  However,  the  color-emotion  association  of  the  two  teenage  (15  year-old) 
subjects.  Subjects  2  and  3,  was  dramatically  different.  This  difference  is  very  interesting 
especially  in  this  age  group.  The  results  are  very  similar  to  Phase  2  of  the  study 
conducted  by  Cuteitta  and  Haggerty  (1987)  discussed  in  Chapter  2  of  this  thesis.  One 
logical  explanation  for  this  difference  is  that  these  two  subjects  are  in  the  middle  of  their 
pubescent  years  and  are  probably  just  beginning  to  get  a  foothold  on  their  emotions. 

While  their  color  choices  are  not  identical  to  each  other  they  are  similar, 
especially  in  the  grouping  of  colors.  In  general  they  selected  the  warmer  colors  to 
represent  the  peaceful  and  calmer  emotions  and  selected  the  cooler  colors  to  represent 
the  energetic  and  happy  emotions.  Even  though  there  were  differences  in  the  other 
subject’s  selection,  they  kept  the  warm  colors  with  the  happy  and  energetic  emotions  and 
the  cooler  colors  with  the  peaceful  and  calmer  emotions. 

This  shows  that,  for  a  large  audience  of  different  age  groups,  a  default  color 
selection  may  not  provide  everyone  with  a  similar  experience. 

For  this  work,  the  difference  in  color  selection  is  interesting,  but  not  a  huge 
factor.  The  color-emotion  association  within  See  Symphony  was  customized  for  each 
subject  based  on  their  selections. 

I  would  have  been  interested  in  some  younger  subjects,  but  it  may  have  been 
difficult  to  explain  some  of  the  emotional  categories  accurately,  as  well  as  getting 


accurate  color  associations. 
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Responses  to  Animation 

The  responses  to  each  piece  of  music  will  be  discussed  individually.  Overall  the 
subjects  followed  the  instructions  well.  Subjects  2  and  3  did  tend  to  get  distracted  at 
times,  but  they  were  still  able  to  provide  worthwhile  feedback. 

24  Caprices  for  Solo  Violin,  Op.  1,  no.  4  in  C  Minor  by  Paganini.  This  piece 
of  music  was  selected  for  the  sad  and  serene  qualities  of  its  first  minute.  When  the 
animation  runs  it  primarily  uses  the  colors  of  the  sad,  calm,  and  peaceful  emotional 
categories.  Other  colors  did  show,  but  their  appearance  was  appropriate.  For  example, 
when  the  tempo  would  increase  for  an  instance  the  color  would  change  to  a  ‘happier’ 
color. 

The  hearing  subjects  agreed  that  the  piece  was  subtle  and  soothing.  It  was  also 
noted  that  there  seemed  to  be  some  struggle  with  in  the  piece.  This  probably  due  to  the 
highly  disjunct  melody. 

The  hearing  subjects  also  agreed  that  the  animation  corresponded  to  the 
animation,  but  they  differed  on  how  well.  Subject  4  noted  that  theme  were  there,  but 
they  did  not  flow  very  well  and  were  confusing.  This  subject  said  that  he  did  not  care  for 
the  music.  This  and  the  fact  that  the  music  is  highly  disjunct  could  be  cause  for  his 
interpretation.  On  the  other  hand  Subject  5  felt  that  the  animation  corresponded  well  to 
the  music.  Subject  5  also  pointed  out  something  very  important.  She  said  the  color 
changes  were  confusing  until  she  relaxed  and  just  listened.  I  believe  for  the  animations 
to  work  well  you  must  watch  them  the  way  you  listen  to  music.  That  is,  pay  close 
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attention,  but  don’t  try  to  analyze  and  dissect  every  instance.  Most  important  of  all, 
relax. 

The  responses  from  the  deaf  subjects  were  also  interesting.  Tables  4.3  and  4.6 
reveal  that  Subject  l’s  response  closely  matched  that  of  the  hearing  subjects.  Subject  2 
and  3  were  less  revealing,  but  informative  nonetheless.  Subject  2  said  the  animation 
portrayed  a  good  feeling,  which  is  close  to  subdued  and  calm.  Subject  3  can  only  hear 
loud  noises,  he  reacted  to  some  of  he  sounds  that  he  could  hear.  As  mentioned  before, 
the  speakers  were  turned  up  to  a  high  volume.  At  some  of  the  peaks  (volume)  in  the 
music  Subject  3  was  visibly  annoyed.  Although  those  were  the  only  sounds  he  could  hear 
he  seemed  very  sensitive  to  them.  I  believe  it  is  comparable  to  sitting  in  a  dark  room  and 
turning  on  a  bright  light. 

Sonata  no.  3  in  C  Major,  Allegro  Assai  by  Bach.  This  piece  was  chosen  for  its 
playful  and  cheerful  melody.  The  animation  primarily  used  colors  of  the  exciting,  happy, 
and  lively  colors.  As  with  the  first  animation,  other  colors  did  show,  but  they 
appropriately  followed  the  mood  of  the  music. 

Table  4.6  reveals  that  subjects  4  and  5  agreed  with  this  assessment  of  the  music. 
Subject  5  gave  a  particularly  in  depth  and  interesting  description.  Subject  5  also 
mentioned  that  the  colors  emphasized  the  happy  feelings  of  the  music.  Subject  4  stated 
that  this  animation  corresponded  to  the  music  much  better  than  the  first  animation.  The 
fact  that  the  music  was  more  pleasant  could  have  affected  his  decision. 

Subject  1  expressed  that  it  made  her  want  to  get  up  and  dance.  Subjects  2  and  3 
both  found  the  animation  for  this  music  exciting.  If  you  compare  the  reactions  of  Subject 


62 


2  to  the  works  by  Paganini  and  Bach,  you  can  see  that  the  animations  are  presenting  the 
correct  emotions  for  the  music.  There  is  an  elevation  of  emotion  between  the  two  pieces 
and  there  is  also  a  corresponding  elevation  in  his  responses. 

Theme  from  JAWS  by  John  Williams.  The  software  is  made  to  handle  solo 
instruments,  specifically  the  violin,  but  I  thought  it  would  be  interesting  to  see  how  it 
worked  with  a  full  orchestra.  One  problem  that  was  very  apparent  was  that  the  software 
has  a  difficult  time  identifying  the  tempo  properly  when  there  are  multiple  instruments 
playing.  It  has  the  most  problem  with  identifying  slow  tempos.  The  software  examines 
the  duration  of  the  most  significant  frequency  to  determine  the  tempo.  If  two  or  more 
instruments  are  playing  it’s  unlikely  that  are  playing  the  same  frequencies  at  the  same 
moment  at  the  same  magnitude. 

With  one  instrument  See  Symphony  measures  the  time  from  the  beginning  of  one 
frequency  to  the  end  of  that  frequency.  In  Figure  4. 1  two  instruments  are  playing 
separate  notes  of  different  duration  and  magnitude.  Instead  of  measuring  the  time  from 
point  A  to  D  for  a  solo  instrument  See  Symphony  now  measures  from  A  to  B,  from  B  to 
C,  and  C  to  D.  See  Symphony  sees  it  as  one  instrument  changing  notes  three  times 
instead  of  two  instruments  playing  one  note.  When  the  tempo  is  analyzed  See  Symphony 
will  determine  a  faster  tempo  than  it  should. 
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The  same  thing  can  happen  in  reverse.  A  high  magnitude,  long  duration 
frequency  can  mask  the  notes  of  another  instrument  playing  at  a  lesser  magnitude. 


Figure  4.2  False  Slow  Tempo  Identification 


Instrument  1 


I  Instrument  2 


In  Figure  4.2,  again  there  are  two  instruments  playing  separate  notes  of  different 
duration  and  magnitude.  This  time  See  Symphony  will  measure  the  duration  from  A  to  B 
and  miss  all  of  the  durations  of  instrument  2.  This  will  falsely  identify  the  tempo  as 
being  slow  when  it  is  actually  fast. 

If  the  tempo  is  identified  incorrectly  then  the  emotion  will  be  displayed 
improperly.  Recall  from  Table  3.1  Emotional  Expressions  Related  to  Musical  Elements 
for  Violin,  that  the  tempo  plays  a  major  role  in  identifying  the  emotion  of  the  music. 
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With  all  this  in  mind  I  did  not  expect  the  theme  from  Jaws  to  be  very  accurate  in 
determining  the  slow  tempos  when  multiple  instruments  were  playing. 

Even  with  these  known  limitations  the  animation  did  very  well  and  the  responses 
reflected  such.  Subjects  4  and  5  both  noted  a  building  of  emotions  from  the  music:  fear, 
anxiety,  and  anticipation.  They  both  agreed  that  the  animation  corresponded  to  the  music 
well.  Both  commented  that  the  same  building  of  emotion  was  evident  in  the  animation. 

All  hearing  impaired  subjects  enjoyed  this  animation  the  most.  Subject  2  thought 
it  was  exciting.  Subject  3  only  noted  that  he  preferred  the  lower  volume  shapes.  At 
lower  volumes  the  shapes  are  smaller,  when  the  volume  builds  the  shape  gets  larger. 
Subject  3  could  see  when  an  unpleasant  sound  was  coming. 

Subject  1  said  that  the  animation  made  her  feel  like  dancing  wildly.  That  feeling 
is  very  apparent  towards  the  end  of  the  piece  when  the  music  becomes  chaotic. 

General  Observations 

While  each  of  the  deaf  subjects  was  watching  the  animations  I  was  watching  them 
for  physical  reactions.  The  most  obvious  reaction  was  to  volume.  A  sudden  increase  in 
the  volume  causes  the  image  on  the  screen  to  grow  rapidly.  When  an  increase  in  volume 
occurred  the  subjects  would  move  their  heads  back  slightly  just  as  a  hearing  person 
would  if  confronted  with  a  sudden  load  noise.  Subject  2  and  3  could  hear  loud  noises, 
but  Subject  3  was  completely  deaf,  so  I  could  be  sure  that  her  reaction  was  from  the 


animation. 
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The  deaf  subjects  also  smiled  often  during  the  animation.  I  assumed  that  they 
were  enjoying  the  animations.  They  all  confirmed  this  suspicion  at  the  end  of  each 
session  by  telling  me  that  they  enjoyed  it  and  would  like  to  see  more. 

I  found  this  test  to  be  very  successful.  It  proves  my  hypothesis  that  the  emotions 
of  music  can  be  automatically  identified  an  displayed  visually.  Although,  See  Symphony 
is  not  perfect,  it  goes  a  long  way  to  proving  what  can  be  done. 

Group  Reactions 

As  mentioned  before,  the  group  observation  was  less  structured  and  less 
controlled  than  the  sessions  in  Phase  I.  I  simply  observed  and  gathered  data  during  and 
after  the  presentation. 

Everyone  enjoyed  the  display,  but  what  was  the  most  interesting  was  that  the 
individuals  who  had  the  best  reaction  to  the  animation  were  closer  to  the  display.  Most 
agreed  that  animation  corresponded  to  the  music,  but  those  who  were  farther  away  were 
not  satisfied  with  changing  of  the  colors  on  the  monitor.  They  mentioned  that  the  color 
change  was  too  rapid  or  did  not  hold  long  enough,  or  was  too  erratic  to  portray  an 
emotion.  Also,  some  individuals  farther  away  from  the  primary  display  preferred  the 
animation  on  the  secondary  display.  The  secondary  display  had  a  slow  refresh  rate  which 
caused  it  to  leave  a  shadow  of  the  previous  images  on  the  screen  for  a  few  milliseconds. 

Discussion 

The  fact  that  those  individuals  closest  to  the  display  had  the  most  positive 
reactions  and  a  more  complete  experience  supports  the  fact  the  animation  is  similar  to 
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music.  As  mentioned  in  Chapter  2,  in  order  to  fully  appreciate  music  you  must  actively 
listen  to  it.  The  degree  to  which  music  affects  us  depends  a  great  deal  on  how  we 
perceive  it.  There  is  a  distinction  between  hearing  and  listening  to  music.  Likewise, 
with  the  animations  it  is  necessary  to  give  it  your  full  attention.  For  the  animations  to  be 
more  effective  one  must  watch  and  absorb  the  animation,  not  just  simply  look. 

Another  reason  that  those  closer  to  the  display  had  a  more  positive  experience 
could  also  be  due  to  the  fact  that  they  were  in  a  direct  line  of  the  soundswaves  from  the 
speakers.  As  with  Phase  I,  the  volume  was  set  rather  loud.  The  individual  closest  to  the 
monitor  would  have  been  able  to  feel  the  music  through  the  sound  waves.  The  vibrations 
of  the  music  emphasizes  the  shapes  used  in  the  animations.  For  example,  a  loud, 
rumbling  low  pitch,  would  produce  a  large,  wide,  vibrating  shape  as  seen  in  Figure  4.3. 
The  ability  to  experience  both  the  vibrations  and  the  animation  would  only  enhance  the 
experience. 


Figure  4.3  Display  Simulation 


Another  point  must  be  made  regarding  the  changing  of  the  colors.  If  you  focus  on 
the  rate  of  which  colors  change  then  you  are  trying  to  force  an  interpretation  of  the 
emotions  that  are  being  displayed.  Rather,  you  should  just  absorb  what  is  happening  and 
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emotions  will  become  evident.  Subject  5  in  Phase  I  noted  that  she  felt  the  color  changes 
were  a  little  confusing  until  she  relaxed  and  just  listened. 

I  received  e-mailed  responses  from  five  members  of  the  Phase  II  group  regarding 
See  Symphony.  Their  complete  messages  can  be  found  in  Appendix  D.  I  inserted  a  note 
at  the  top  of  each  letter  stating  the  individuals  relative  position  to  the  primary  display,  if 
known. 


CHAPTER  5 


CONCLUSIONS  AND  FURTHER  RESEARCH 

Conclusions 

Music  is  capable  of  representing  a  wide  range  of  feelings  and  emotions.  The 
emotions  and  feeling  are  complex  in  their  form  and  they  develop  over  time,  but  they  are 
comprised  of  basic  elements:  pitch,  volume,  and  tempo.  These  three  properties  have 
allowed  me  to  identify  the  emotions  that  are  portrayed  by  music. 

Color  can  have  a  deep  effect  on  our  moods.  Colors  are  consistently  associated 
with  emotions  and  feelings.  Color  is  a  natural  visual  means  of  expressing  emotion.  I 
have  used  color  to  represent  the  emotions  and  feeling  of  music. 

Pitch,  volume,  and  tempo  also  correlate  to  physical  shapes  and  movement.  I  have 
incorporated  these  elements  into  the  animations. 

The  subset  of  our  population  who  are  not  able  to  enjoy  music  as  it  was  meant  to 
be  enjoyed  will  now  be  able  to  see  what  was  not  fully  available  to  them.  Music  is  too 
beautiful  to  limit  its  audience.  See  Symphony  translates  music  into  a  visual  form 
allowing  the  emotions  of  music  to  be  portray  visually.  Music  can  be  enjoyed  and 
appreciated  via  other  senses  besides  hearing. 
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I  set  out  to  show  that  it  is  possible  to  automatically  identify  the  emotions 
portrayed  by  music  and  display  those  same  emotions  visually.  I  have  accomplished  this 
goal. 

Further  Research 

The  first  thing  that  needs  to  be  accomplished  is  to  use  a  larger  sample  size  of  deaf 
subjects.  See  Symphony  seemed  to  be  heading  in  the  right  direction  with  the  limited 
number  of  subjects  used  in  this  study,  but  a  more  statistically  significant  sample  size 
must  be  used  to  positively  confirm  the  results. 

Solo  violin  was  used  because  of  its  obvious  ability  to  portray  emotions,  its 
popularity,  and  it  wide  frequency  range.  The  next  step  is  to  incorporate  other 
instruments.  The  primary  area  of  inspection  when  dealing  with  other  instruments  must 
be  their  frequency  range.  I  based  the  middle  of  the  violin’s  range  on  A4.  As  noted  in 
Chapter  2,  not  all  instrument  include  A4  in  their  frequency  range.  Should  the  frequency 
of  a  given  note  be  considered  in  absolute  or  relative  terms.  In  other  words,  should  a  note 
be  determined  high,  medium,  or  low  based  on  where  it  falls  among  the  frequencies 
achievable  by  all  instruments  or  should  it  be  measured  based  on  the  frequency  range  of 
the  instrument  playing  the  note? 

The  timbre  of  an  instrument  becomes  an  important  factor,  especially  when  that 
instrument  is  compared  to  others.  Timbre  was  briefly  mentioned  in  this  study,  but  not 
dealt  with  since  I  was  only  concerned  with  the  violin.  Do  some  instruments  typically 
sound  happier  than  others?  Will  different  combinations  of  instruments  change  the 
perceived  timbre  of  another? 
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Several  people  who  viewed  See  Symphony  mentioned  that  they  preferred  the 
display  on  the  notebook  computer.  The  notebook’s  display  has  a  visual  memory  due  to 
its  relatively  slow  refresh  rate.  When  an  image  is  displayed,  the  previous  image  is  still 
visible  for  a  few  milliseconds.  This  gives  the  previous  images  the  illusion  of  fading  away 
instead  of  a  crisp  transition  as  it  would  be  on  a  standard  computer  monitor.  Would  a 
smoother  transition  further  enhance  the  portrayal  of  emotions  in  the  animation?  If  all  the 
animations  used  a  fading  transition  would  it  smooth  out  sharp  music?  It  would  probably 
enhance  the  experience  in  some  cases,  but  not  properly  represent  the  music  in  others. 
There  would  need  to  be  a  way  to  determine  the  type  of  transition  to  use  in  a  given 
situation. 

What  I’ve  accomplished  is  to  allow  hearing  impaired  individuals  to  see  a  solo. 
When  the  same  work  is  done  for  all  orchestral  instruments,  then  the  Deaf  will  truly  be 


able  to  see  a  symphony. 
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See  Symphony  Description 


What  is  See  Symphony? 

See  Symphony  is  the  automatic  identification  and  visualization  of  emotions 
in  music  through  computer  software.  The  goal  is  to  provide  an  additional 
way  to  enjoy  music  for  those  with  hearing  impairments. 


How  Does  See  Symphony  Work? 

See  Symphony  listens  to  music  by  way  of  CD  or  microphone  through  a 
computer.  The  music  is  analyzed  up  to  30  times  a  second  for  pitch,  volume, 
and  tempo.  Based  on  these  properties  See  Symphony  determines  the 
emotions  being  portrayed  by  the  music  and  produces  a  display  to  the  screen. 


How  is  the  Emotion  of  Music  Displayed? 

See  Symphony  uses  colors,  shapes,  and  movement  to  display  the  emotions  of 
music.  The  colors  are  those  of  the  standard  color  wheel:  red,  orange,  yellow, 
green,  blue,  and  purple.  The  default  emotional  category  that  each  color  is 
linked  to  is  based  on  research,  but  the  color-emotion  relationship  may  be 
changed  according  to  the  user’s  preferences. 

The  ellipse  is  the  basic  shape  used  in  the  See  Symphony  animations.  The 
louder  the  music  is  the  larger  the  ellipse  becomes.  The  reverse  is  also  true,  as 
the  volume  becomes  more  quiet  the  ellipse  will  shrink  in  size.  The  shape 
also  reacts  to  pitch.  A  low  pitch  will  cause  the  shape  to  flatten  out,  while  a 
high  pitch  will  cause  the  shape  to  tighten  up. 


APPENDIX  B 


COVER  LETTER  AND  QUESTIONNAIRE  FOR  HEARING- 
IMPAIRED  SUBJECTS  IN  RESEARCH  STUDY 
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See  Symphony  Participant  Instructions  I 


Thank  you  for  participating  in  the  See  Symphony  project. 

The  purpose  of  See  Symphony  is  to  provide  another  way  to  enjoy  music. 
More  specifically,  it  is  designed  to  allow  those  with  hearing  impairments  to 
more  fully  experience  the  emotions  that  music  portrays  through  animation 
and  colors. 

You  will  be  asked  to  complete  a  questionnaire.  The  questionnaire  is  used  to 
gather  some  basic  information  about  your  hearing  impairment  (Section  I)  as 
well  determining  how  you  associate  colors  with  emotions  and  feelings 
(Section  II). 

After  completing  Sections  I  and  II  of  the  questionnaire  you’ll  be  asked  to 
watch  three  separate  animations  of  musical  works.  The  first  two  animations 
will  be  violin  solos  while  the  last  will  be  a  complete  orchestra.  The  music 
will  be  playing  simultaneously  with  the  animations  through  speakers  near  the 
computer  screen.  You  should  be  able  to  feel  the  vibrations  of  the  music 
through  the  table  where  you’ll  be  seated.  You  may  listen  to  the  music 
through  headphones  if  you  prefer. 

Please  focus  all  of  your  attention  on  the  animations.  If  there  is  anything 
distracting  you  please  let  us  know.  At  the  end  of  each  animation  you  will  be 
asked  to  describe  what  emotions  you  felt  were  portrayed  by  the  animation. 
You  will  also  be  able  to  add  any  additional  comments  about  each  animation. 

When  all  of  the  animations  are  complete  feel  free  to  add  any  additional 
overall  comments  about  the  See  Symphony  project. 

If  you  wish  to  be  informed  concerning  further  work  on  this  project  please 
enter  your  contact  information  on  the  See  Symphony  Contact  Sheet. 


Thank  You  for  your  help! 
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Sec  Symphony  Questionnaire  I 


Thank  you  for  taking  the  time  in  assisting  with  this  study.  Please  answer  the  following 
questions  as  accurately  as  possible.  You  may  continue  on  the  back  if  you  need  more 
space  for  your  response. 

Section  I  . 


1)  Age?  _ 

2)  What  is  your  level  of  hearing  impairment?  (circle  one): 

Partial  (Please  answer  Question  2. 1  below) 

Full  (Skip  to  Question  3) 

2.1)  If  you  have  partial  hearing  impairment,  please  describe  (what  sounds  can 
you  hear?): 


3)  Have  you  been  hearing  impaired  from  birth?  (circle  one): 

YES  (Skip  to  Question  4) 

NO  (Please  answer  Question  3.1) 

3.1)  At  what  age  did  you  lose  your  hearing? _ 

4)  Do  you  use  any  devices  to  aid  your  hearing?  (circle  one): 

YES  (Please  answer  Questions  4. 1  and  4.2) 

NO  (Skip  to  Question  5) 

4.1)  What  device  do  you  use? _ 

4.2)  In  what  way  does  it  improve  your  hearing? 


5)  Do  you  have  any  experience  with  music  (listening,  playing,  feeling)?  (circle  one): 

YES  (Please  go  to  5.1) 

NO  (Skip  to  Question  6) 


5.1)  Please  explain  your  musical  experience.  Also  include  your  musical 
preferences: 
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6)  Have  you  been  involved  in  any  other  studies  involving  music  and  hearing  impairment? 
(circle  one): 

YES  (Please  go  to  6.1) 

NO  (Skip  to  Section  II) 

6.1)  Please  explain  the  previous  study(ies)  in  which  you  were  involved: 


Section  II 


Select  the  color  which  you  feel  most  closely  portrays  the  emotions/feelings  in  each  given 
category.  Please  draw  a  line  from  a  color  to  an  emotion/feeling  category  to  indicate  your 
selection.  Use  each  color  only  once. 


COLORS 

Red 

Orange 

Yellow 

Green 

Blue 

Purple 


EMOTIONS/FEELINGS 


Exciting,  Passionate, 
Stimulating _ 


Lively,  Energetic,  Jovial 


Happy,  Cheerful,  Fun 


Refreshing,  Peaceful 


Subduing,  Sober,  Calm 


Sad,  Mournful,  Royal 
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Section  III 


Music  1 :  24  Caprices  for  Solo  Violin,  Op.  1,  No.  4  in  C  Minor  -  Nicolo  P aganini 

Please  describe  the  emotions  or  feeling  that  you  think  the  animation  displayed: 

Additional  Comments: 

Music  2:  Sonata  No.  3  in  C  Major,  Allegro  Assai  -  J.  S.  Bach 

Please  describe  the  emotions  or  feeling  that  you  think  the  animation  displayed: 

Additional  Comments: 

Music  3:  Theme  from  JAWS  -  John  Williams  and  the  Boston  Pops  Orchestra 

Please  describe  the  emotions  or  feeling  that  you  think  the  animation  displayed: 

Additional  Comments: 

APPENDIX  C 

COVER  LETTER  AND  QUESTIONNAIRE  FOR  HEARING 
SUBJECTS  IN  RESEARCH  STUDY 
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See  Symphony  Participant  Instructions  II 


Thank  you  for  participating  in  the  See  Symphony  project. 

The  purpose  of  See  Symphony  is  to  provide  another  way  to  enjoy  music. 

More  specifically,  it  is  designed  to  allow  those  with  hearing  impairments  to 
more  fully  experience  the  emotions  that  music  portrays  through  animation 
and  colors. 

You  will  be  asked  to  complete  a  questionnaire.  The  questionnaire  is  used  to 
gather  some  basic  information  about  you  (Section  I)  as  well  determining  how 
you  associate  colors  with  emotions  and  feelings  (Section  II). 

After  completing  Sections  I  and  II  of  the  questionnaire  you’ll  be  asked  to 
watch  and  listen  to  three  separate  animations  of  musical  works.  The  first  two 
animations  will  be  violin  solos  while  the  last  will  be  a  complete  orchestra. 

The  music  will  be  playing  simultaneously  with  the  animations  through 
speakers  near  the  computer  screen.  You  may  listen  to  the  music  through 
headphones  if  you  prefer. 

Please  focus  all  of  your  attention  on  the  animations.  If  there  is  anything 
distracting  you  please  let  us  know.  At  the  end  of  each  animation  you  will  be 
asked  to  describe  what  emotions  you  felt  were  portrayed  by  the  music  and 
how  well  the  animations  corresponded  to  the  music.  You  may  also  add  any 
additional  comments  about  each  animation. 

When  all  of  the  animations  are  complete  feel  free  to  add  any  additional 
overall  comments  about  the  See  Symphony  project. 

If  you  wish  to  be  informed  concerning  further  work  on  this  project  please 
enter  your  contact  information  on  the  See  Symphony  Contact  Sheet. 


Thank  You  for  your  help! 
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See  Symphony  Questionnaire  II 


Thank  you  for  taking  the  time  in  assisting  with  this  study.  Please  answer  the  following 
questions  as  accurately  as  possible.  You  may  continue  on  the  back  if  you  need  more 
space  for  your  response. 

'Section  I  ...  .  1  ■•••A-.'  .'V 

1)  Age?  _ 

2)  Do  you  have  any  experience  with  music  (listening,  playing,  etc.)?  (circle  one): 

YES  (Please  go  to  2.1) 

NO  (Skip  to  Question  3) 

2.1)  Please  explain  your  musical  experience.  Also  include  your  musical 
preferences: 


3)  Have  you  been  involved  in  any  other  studies  involving  music  and  hearing  impairment? 
(circle  one): 

YES  (Please  go  to  3 . 1 ) 

NO  (Skip  to  Section  II) 

3.1)  Please  explain  the  previous  study(ies)  in  which  you  were  involved: 
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Section  II 


Select  the  color  which  you  feel  most  closely  portrays  the  emotions/feelings  in  each  given 
category.  Please  draw  a  line  from  a  color  to  an  emotion/feeling  category  to  indicate  your 
selection.  Use  each  color  only  once. 


COLORS 

Red 

Orange 

Yellow 

Green 

Blue 

Purple 


EMOTIONS/FEELINGS 


Exciting,  Passionate, 
Stimulating _ 


Lively,  Energetic,  Jovial 


Happy,  Cheerful,  Fun 


Refreshing,  Peaceful 


Subduing,  Sober,  Calm 


Sad,  Mournful,  Royal 
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Section  HI 


Music  1:  24  Caprices  for  Solo  Violin,  Op.  1,  No.  4  in  C  Minor  -  Nicolo  Paganini 

Please  describe  the  emotions  or  feelings  that  you  think  the  music  portrayed: 

How  did  the  animation  correspond  to  the  emotions  and  feelings  of  the  music: 

Additional  Comments: 

Music  2:  Sonata  No.  3  in  C  Major,  Allegro  Assai  -  J.  S.  Bach 

Please  describe  the  emotions  or  feelings  that  you  think  the  music  portrayed: 

How  did  the  animation  correspond  to  the  emotions  and  feelings  of  the  music: 

Additional  Comments: 

Music  3:  Theme  from  JAWS  -  John  Williams  and  the  Boston  Pops  Orchestra 

Please  describe  the  emotions  or  feelings  that  you  think  the  music  portrayed: 

How  did  the  animation  correspond  to  the  emotions  and  feelings  of  the  music: 

Additional  Comments: 

APPENDIX  D 


MESSAGES  FROM  GROUP  SUBJECTS 
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*  Subject  Position:  in  front  of  primary  display,  but  7-8  feet  away. 


From:  Bob  Gattis 
To:  John  Billups 

Subject:  See  Symphony  Demo  Feedback 
Date:  Friday,  November  14, 1997  4:46  AM 

Hi  Jay, 

Sorry  this  is  a  little  late... 

Your  demo  was  excellent  - 1  liked  the  simplicity  of  the  ellipse  you  chose  to  animate  the 
music's  features.  It  was  surprising  how  much  information  a  single  shape  could  convey. 
The  real-time  nature  of  the  sound  processing  was  impressive,  although  I  would  like  to  see 
you  experiment  with  some  form  of  signal  averaging,  to  smooth  out  the  visual  response  to 
the  music.  I  found  the  rapid  visual  transitions  of  both  color  and  shape  somewhat 
distracting. 

As  far  as  visually  representing  emotions,  I  have  to  confess  that  the  demo  was  a  mixed 
success.  For  me,  the  rapid  changes  in  the  visual  display  for  some  music  worked  against 
creating  a  sustained  emotional  "feel".  In  addition,  I  would  suggest  you  try  some 
additional  types  of  musical  input  for  emotional  content  testing.  Specifically,  you  might 
try  organ  music,  such  as  funeral  dirges,  for  the  sad  emotion  and  a  flute  solo,  for  a  lighter, 
happier  emotion.  This  would  also  separate  the  musical  tonalities  -  pitch  as  well  as 
volume.  The  deaf  or  hard  of  hearing  person  would  also  probably  be  able  to  sense  the 
difference  between  these  musical  selections  through  the  different  vibrations  produced. 

In  summary,  I  think  you  did  a  superb  job  of  attacking  the  problem  and  have  a  great 
foundation  for  further  testing  and  development.  What  are  you  waiting  for?  Get  back  to 
work  and  quit  reading  your  email! 

Best  wishes. 

Bob 


Bob  Gattis  -  Meeting  the  Challenge,  Inc. 
3630  Sinton  Rd.,#103 
Colorado  Springs,  CO  80907 
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*  Subject  Position:  directly  in  front  of  primary  display. 

From:  Steve  Stock 

To:  John  Billups 

Subject:  Demo  comments 

Date:  Thursday,  November  13, 1997  20:36  PM 


Hi  Jay 

This  is  Steve  from  Meeting  the  Challenge,  with  a  couple  of  comments  on  your  recent 
demo.  First,  if  you  remember,  I  found  myself  saying  after  the  demo  "It  would  be  great  to 
hear  that  without  the  sound,  "-somewhat  of  an  oxymoron,  but  revealing  in  its  own  way. 
Secondly,  it  seems  to  me  their  is  no  end  to  the  research  and  testing  that  could  be  done 
with  your  system.  It  would  be  interesting  to  see  how  consistently  (and  quickly)  people 
with  deafness  could  identify  a  piece  of  music  after  becoming  somewhat  familiar  with  the 
"screen  dance"  of  images  that  your  system  provides.  Inasmuch  as  hearing  people  enjoy 
listening  to  music,  do  non-hearing  people  enjoy  watching  it?  Would  non-hearing  people 
discriminate  between  'good'  music  and  'bad'  music  in  the  same  way  hearing  people  do? 
For  that  matter,  does  'good',  or  popular  music  consistently  remain  good  when  transformed 
to  the  computer  screen?  Anyhow,  good  luck  with  the  thesis,  graduation,  etc.  I  received 
my  graduate  degree  from  UCCS,  and  it  was  a  great  experience.  —Steve 


88 


*  Subject  Position:  closer  to  secondary  display. 


From:  Randy  Dipner 

To:  John  Billups 

Subject:  See  Symphony 

Date:  Tuesday,  November  11, 1997  16:30  PM 

Jay, 

Thank  you  for  the  opportunity  to  be  a  part  of  your  Masters  project.  We  hope  your 
interactions  with  MTC  were  as  valuable  to  you  as  meeting  you  was  to  us. 

I  was  quite  impressed  with  the  results  of  your  work.  For  me  the  images  seemed  to  be  very 
much  connected  to  the  music.  I  could  directly  connect  the  size,  proportions  and 
roughness  of  the  images  to  the  volume,  pitch  and  dissonance  of  the  music.  I  did  feel  that 
the  mood  of  the  music  was  less  well  connected  to  the  color  of  the  images.  It  seems  to  me 
that  mood  in  music  is  a  longer  duration  impression  built  up  over  several  bars  rather  than 
the  very  quick  changes  implied  by  the  rapidly  changing  colors.  I  wonder  whether  some 
method  of  averaging  over  time  could  provide  this  rather  smoother  transition  from  one 
feeling  to  another.  I  was  also  more  pleased  with  the  impression  of  the  LCD  display  than 
the  monitor.  The  latency  of  the  images  on  the  LCD  provided  more  of  a  clear  sense  of  the 
music  for  me  than  the  clean  break  from  image  to  image  on  the  CRT  display.  I  can 
imagine  a  very  impressive  display  with  each  instrument  separately  mic'ed  and  the  images 
displayed  in  some  form  of  overlapping  presentation. 

Congratulations  on  the  success  of  your  research.  I  hope  you  will  stay  in  touch  as  you 
move  forward  with  this  project. 

Please  feel  free  to  use  anything  I  have  said  here  at  your  discretion. 

Sincerely, 

Randy  Dipner 
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*  Subject  Position:  not  recorded. 

From:  Mark  Snow 

To:  John  Billups 

Subject:  See  Symphony  test 

Date:  Monday,  November  10, 1997  12:19  PM 

Greetings  Jay: 

Thanks  for  the  opportunity  to  take  part  in  the  review  of  See  Symphony.  It  was  great  to 
see  your  enthusiasm  and  dedication  toward  something  that  will  be  of  benefit  to  thousands 
of  people. 

Your  demonstration  reminded  me  of  the  vivid  colors  and  sweeping  melodies  that  make 
up  some  of  the  segments  of  "Fantasia".  I  think  it  is  fair  to  say  you  have  taken  current 
technology  and  made  it  both  useful  and  entertaining. 

Best  of  luck  in  the  future  with  See  Symphony! 


Mark  Snow 
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*Subject  Position,  not  recorded. 

From:  Pat  Going 

To:  John  Billups 

Subject:  See  Symphony 

Date:  Monday,  November  10, 1997  14:34  PM 

Pat  Going 

Thank  you,  Jay,  for  your  demonstration  last  week.  As  you  know,  people  who  are  deaf  or 
hard-of-hearing  constitute  almost  50%  of  the  total  number  of  people  with  disabilities  in 
the  U.S.  Incorporating  visual  representation  -  like  your  program  -  into  concerts,  etc.  will 
have  a  tremendous  appeal  to  many  people.  Be  sure  and  come  back  as  you  continue  to 
make  refinements  to  this  ambitious  and  exciting  project. 

Patrick  Going,  Project  Director, 

Rocky  Mountain  ADA  Technical  Assistance  Center. 


*  Subject  Position:  close,  just  to  right  of  primary  display. 

From:  Alan 

To:  John  Billups 

Subject:  Symphony  response 

Date:  Thursday,  November  20, 1997  12:26  PM 

Alan  J  Ocken 

Meeting  The  Challenge,  Inc. 

I  would  like  to  say  first  of  all  that  I  truly  enjoyed  the  presentation  that 
was  made  and  feel  that  software  such  as  this  has  a  place  with  hearing  folks 
as  well  as  deaf.  It  took  me  back  to  the  days  when  I  would  stair  for  hours 
at  my  graphic  equalizer.  Walt  Disney  made  a  ton  of  money  from  Fantasia  the 
movie.  A  scene  in  that  movie  had  multi-colored  waves  that  were 
choreographed  to  evoke  an  emotional  response.  This  type  of  technology  will 
create  the  ability  to  have  real  time  (not  choreographed)  images  to 
accompany  music.  The  issue  of  changing  colors  and  patterns  at  too  rapid  of 
a  rate:  It  is  a  gut  feeling  -  totally  unsubstantiated  -  that  a  hearing 
person  and  more  over  a  deaf  person's  brain  is  capable  of  sampling  at  a  very 
rapid  pace  and  then  sub-consciously  assembling  these  samples  into  an 
emotional  response.  After  all  is  said  a  done,  music  is  a  language  that 
conveys  an  emotional  message.  The  Bible  itself  speaks  about  music  in  well 
over  900  separate  verses.  If  music  is  a  language  then  it  must  be  assumed 
that  this  language  may  be  conveyed  in  multiple  ways.  We  have  written  text, 
verbal  language,  ASL,  and  Braille  that  convey  the  language  we  call  English. 

Our  mind  must  learn  to  assemble  separate  bits  of  information  or  samples 
and  then  assemble  those  samples  in  an  understandable  way.  Stephen  King, 
Louis  Farrakhan,  William  Shakespeare,  and  Jesus  all  used  the  same  set  of 
basic  tools  -  languages  to  communicate  drastically  different  messages. 

The  words  remained  consistent,  it  is  how  those  words  are  arranged  and 
crafted  then  interpreted  that  solicit  the  desired  emotion.  Our  incredible 
brains  make  this  possible.  The  use  of  colors  and  shapes  to  convey  a 
musical  message  is  exciting  bit  of  research  that  should  be  followed  up  on. 


